# Are the shortcomings of neural networks diminishing?

Having worked with neural networks for about half a year, I have experienced first-hand what are often claimed as their main disadvantages, i.e. overfitting and getting stuck in local minima. However, through hyperparameter optimization and some newly invented approaches, these have been overcome for my scenarios. From my own experiments:

• Dropout seems to be a very good regularization method.

• Batch normalization eases training and keeps signal strength consistent across many layers.

I have experimented with scikit-learn's implementation of SVM alongside my experiments with neural networks, but I find the performance to be very poor in comparison, even after having done grid-searches for hyperparameters. I realize that there are countless other methods, and that SVM's can be considered a sub-class of NN's, but still.

So, to my question:

With all the newer methods researched for neural networks, have they slowly - or will they - become "superior" to other methods? Neural networks have their disadvantages, as do others, but with all the new methods, have these disadvantages been mitigated to a state of insignificance?

I realize that oftentimes "less is more" in terms of model complexity, but that too can be architected for neural networks. The idea of "no free lunch" forbids us to assume that one approach always will reign superior. It's just that my own experiments - along with countless papers on awesome performances from various NN's - indicate that there might be, at the least, a very cheap lunch.

• There are several related questions on this site: this, this, this. There are probably other posts. – nbro Jan 18 at 12:11

Neural Networks have other short comings as well.

1. It takes much longer and far more resources to train a neural network than something like a random forest. So if you need speed of training or are resource constrained in anyway, you probably should not look at Neural Networks first. Evaluation of a trained deep NN can be much more expensive than competing techniques too.
2. The effort involved in learning how to architect and train a NN is still much higher than competing methods, like an SVM. People who are just starting out in Data Science should probably use other techniques to learn about the nuances of fitting data before getting involved in neural networks. And although simple NNs with only one or two hyperparameters are often available in many data science libraries, they don't perform any better than other techniques so are just another ML black-box technique really.
3. While we have made a lot of progress in understanding how neural networks do their magic, they are still less accessible and dissectible than most competing methods. So while NNs might solve the problem, they might not give you as many insights as easily as other techniques do.

Looking forward to what other people have to say here.

• Good points, although some of the performance issues can be mitigated by GPU's, I think. Additionally, I can warmly recommend Keras, as it enables very easy composition of basic MLPs. Although, as you say, when using a very high level library such as SciKit Learn, it is a bit hard to get good performance, as the hyperparameters are a bit too superficial, and they don't provide some of the newer methodologies. – Alexander C. Harrington Jan 10 '17 at 10:55
• Yes, but even with massive numbers of GPUs, they are still a lot slower than competing techniques. DNN training is really heavy, I am working on projects at a very big tech company where they routinely talk about days of training of DNNs on very big GPU clusters devoted to these kinds of tasks alone. You don't see that kind of computation devoted to random forest based algorithems AFAIK. – Mike Wise Jan 10 '17 at 10:59
• I see. But I guess this is in exchange for a vastly superior performance from DNNs? Or perhaps the margin that makes a difference? – Alexander C. Harrington Jan 10 '17 at 11:01
• Well, we know DNNs are potentially capable of human level intelligence :) - although we don't know exactly how yet. Concretely, today, we know they give us the best results for tasks like complex image recognition, decoding sound, intent inference in a conversation, and a few others. – Mike Wise Jan 10 '17 at 11:03

Just to add to what has been said in @MikeWise's brilliant answer,

• All things equal, deep learning models generally rank supreme when compared to other algorithms as the size of the dataset increases:

• Like everything, it all boils down to the dataset at hand, neural networks are good on other datasets but at the same time, they will be bad on other datasets. When it comes to unstructured problems(e.g visuals, text, sound), at this time neural networks appear to be the best algorithm. That said, when it comes to structured data, a quick scan at the type of algorithm used to win online data science competitions reveal that, the so called machine learning algorithms such as XGboost rank supreme.

• When it comes to other models, feature engineering plays a big role in the algorithm's efficiency. Feature Engineering is generally a tricky thing to do and do right. Deep learning algorithms don't require as much feature engineering(if any at all) compared to other algorithms, in fact they learn features on their own.

• If the Google guys say they didn't see deep learning coming who is to rule out the possibility of some so called machine learning algorithm coming out and taking over the world by storm?

• Here is a Poll on what data scientist said when asked: if deep learning match the hype in real world application?.

• Even some of the popular deep learning applications like Google's AlphaGo aren't 100% deep learning, instead they are part deep learning, part good old "machine learning". My 2 cent is, maybe we shouldn't rule out other machine learning algorithms yet.