In my opinion, deep learning algorithms and models (that is, multi-layer neural networks) are more sensitive to overfitting than machine learning algorithms and models (such as the SVM, random forest, perceptron, Markov models, etc.). They are capable of learning more complex patterns. At least that's how I look at it. Still, I and my colleagues disagree about this and I cannot really find any information about this. My colleagues say that deep learning algorithms are hardly vulnerable to overfitting.

Are there statements (or opinions) about this aspect?

  • 1
    $\begingroup$ I've not seen significant research done on this, as it is an unusual and specific thing to compare between these models. But at a glance, I would agree that a CNN for example would be much more prone to over-fitting than an SVM, especially without the significant effort put in to ensure it doesn't over fit (dropout, regularisation and so on). $\endgroup$
    – Recessive
    Nov 8 '19 at 23:56

Your reasoning isn't wrong. Deep Neural Networks (DNNs) have a much larger capacity than simpler ML algorithms (excluding NNs) and can easily memorize even a very complex dataset and overfit.

DNNs, however, are so effective because they usually are applied on tasks that are harder, so it's not as easy to overfit. For example an image classifier might be trained on a dataset with millions of images; a task much harder to overfit on.

In cases where this isn't possible (e.g. an image classification task with a couple thousand images), transfer learning is used. You can initialize your weights from a model pre-trained on a large dataset, use its already-trained feature extraction layers and simply fine tune the last layer.

Data augmentation also helps a lot here, which effectively increases size of the training set and discourages the DNN from memorizing the samples. It is so effective that it is used even in large datasets, where it is harder to overfit.

Additionally, DNNs employ several methods to prevent them from overfitting. The most prominent of these is dropout, which is a very effective regularizer. Batch Normalization has also proven an effective regularizer. SGD allows you to explore more parameters than GD, which also is effective against overfitting. Finally, early stopping and parameter norm penalties aren't uncommon in DNNs.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.