Issues with and alternatives to Deep Learning approaches?

Question

Over the last 50 years, the rise/fall/rise in popularity of neural nets has acted as something of a 'barometer' for AI research.

It's clear from the questions on this site that people are interested in applying Deep Learning (DL) to a wide variety of difficult problems.

I therefore have two questions:

1. Practitioners - What do you find to be the main obstacles to applying DL 'out of the box' to your problem?
2. Researchers - What techniques do you use (or have developed) that might help address practical issues? Are they within DL or do they offer an alternative approach?

Answer 1

To summarize, There are two major issues in applied Deep Learning.

• The first being that computationally , it's exhaustive. Normal CPU's require a lot of time to perform even the basic computation/training with Deep Learning. GPU's are thus recommended however, even they may not be enough in a lot of situations. Typical deep learning models don't support the theoretical time to be in Polynomials. However, if we look at the relatively simpler models in ML for the same tasks, too often we have mathematical guarantees that training time required for such simpler Algorithms is in Polynomials. This, for me, at least is probably the biggest difference.

  There're solutions to counter this issue, though. One main approach being is to optimize DL Algorithms to a number of iterations only (instead of looking at the global solutions in practice, just optimize the algorithm to a good local solution, whereas the criterion for "Good" is defined by the user).

• Another Issue which may be a little bit controversial to young deep learning enthusiasts is that Deep Learning algorithms lack theoretical understanding and reasoning. Deep Neural Networks have been successfully used in a lot of situations including Hand writing recognition, Image processing, Self Driving Cars, Signal Processing, NLP and Biomedical Analysis. In some of these cases, they have even surpassed humans. However, that being said, they're not under any circumstance, theoretically as sound as most of Statistical Methods.

  I will not go into detail , rather I leave that up to you. There're pros and cons for every Algorithm/methodology and DL is not an exception. It's very useful as has been proven in a lot of situations and every young Data Scientist must learn at least the basics of DL. However, in the case of relatively simple problems, it's better to use famous Statistical methods as they have a lot of theoretical results/guarantees to support them. Furthermore, from learning point of view, it's always better to start with simple approaches and master them first.

Answer 2

I have very little experience with ML/DL to call myself either practitioner, but here is my answer on the 1st question:

At its core DL solves well the task of classification. Not every practical problem can be rephrased in terms of classification. Classification domain needs to be known upfront. Although the classification can be applied to any type of data, it's necessary to train the NN with samples of the specific domain where it'll be applied. If the domain is switched at some point, while keeping the same model (NN structure), it'll have to be retrained with new samples. Furthermore, even the best classifiers have "gaps" - Adversarial Examples can be easily constructed from a training sample, such that changes are imperceptible to human, but are misclassified by the trained model.

Answer 3

Question 2. I am researching whether Hyper dimensional computing is an alternative to Deep Learning. Hyper-D uses very long bit vectors (10,000 bits) to encode information. The vectors are random and as such they are approximately orthogonal. By grouping and averaging a collection of such vectors a "set" can be formed and later queried to see if an unknown vector belongs to the set. The set can be considered a concept or a generalize image, etc. Training is very fast as is recognition. What needs to be done is simulate the domains in which Deep Learning has been successful and compare Hyper-D with it.

Answer 4

From a mathematics point of view one of the major issues in deep networks with several layers are vanishing or unstable gradients. Each additional hidden layer learns significantly slower, almost nullifying the benefit of the additional layer.

Modern deep learning approaches can improve this behavior, but in simple, old fashioned neural networks this is a well known issue. You can find a well written analysis here for deeper study.