Are the shortcomings of neural networks diminishing?

Question

Having worked with neural networks for about half a year, I have experienced first-hand what are often claimed as their main disadvantages, i.e. overfitting and getting stuck in local minima. However, through hyperparameter optimization and some newly invented approaches, these have been overcome for my scenarios. From my own experiments:

• Dropout seems to be a very good regularization method.

• Batch normalization eases training and keeps signal strength consistent across many layers.

• Adadelta consistently reaches very good optima

I have experimented with scikit-learn's implementation of SVM alongside my experiments with neural networks, but I find the performance to be very poor in comparison, even after having done grid-searches for hyperparameters. I realize that there are countless other methods, and that SVM's can be considered a sub-class of NN's, but still.

So, to my question:

With all the newer methods researched for neural networks, have they slowly - or will they - become "superior" to other methods? Neural networks have their disadvantages, as do others, but with all the new methods, have these disadvantages been mitigated to a state of insignificance?

I realize that oftentimes "less is more" in terms of model complexity, but that too can be architected for neural networks. The idea of "no free lunch" forbids us to assume that one approach always will reign superior. It's just that my own experiments - along with countless papers on awesome performances from various NN's - indicate that there might be, at the least, a very cheap lunch.

Answer 1

Neural Networks have other short comings as well.

1. It takes much longer and far more resources to train a neural network than something like a random forest. So if you need speed of training or are resource constrained in anyway, you probably should not look at Neural Networks first. Evaluation of a trained deep NN can be much more expensive than competing techniques too.
2. The effort involved in learning how to architect and train a NN is still much higher than competing methods, like an SVM. People who are just starting out in Data Science should probably use other techniques to learn about the nuances of fitting data before getting involved in neural networks. And although simple NNs with only one or two hyperparameters are often available in many data science libraries, they don't perform any better than other techniques so are just another ML black-box technique really.
3. While we have made a lot of progress in understanding how neural networks do their magic, they are still less accessible and dissectible than most competing methods. So while NNs might solve the problem, they might not give you as many insights as easily as other techniques do.

Looking forward to what other people have to say here.

Answer 2

Just to add to what has been said in @MikeWise's brilliant answer,

• All things equal, deep learning models generally rank supreme when compared to other algorithms as the size of the dataset increases:

  

• Like everything, it all boils down to the dataset at hand, neural networks are good on other datasets but at the same time, they will be bad on other datasets. When it comes to unstructured problems(e.g visuals, text, sound), at this time neural networks appear to be the best algorithm. That said, when it comes to structured data, a quick scan at the type of algorithm used to win online data science competitions reveal that, the so called machine learning algorithms such as XGboost rank supreme.

• When it comes to other models, feature engineering plays a big role in the algorithm's efficiency. Feature Engineering is generally a tricky thing to do and do right. Deep learning algorithms don't require as much feature engineering(if any at all) compared to other algorithms, in fact they learn features on their own.

• If the Google guys say they didn't see deep learning coming who is to rule out the possibility of some so called machine learning algorithm coming out and taking over the world by storm?

• Here is a Poll on what data scientist said when asked: if deep learning match the hype in real world application?.

• Even some of the popular deep learning applications like Google's AlphaGo aren't 100% deep learning, instead they are part deep learning, part good old "machine learning". My 2 cent is, maybe we shouldn't rule out other machine learning algorithms yet.