Over the last 50 years, the rise/fall/rise in popularity of neural nets has acted as something of a 'barometer' for AI research.

It's clear from the questions on this site that people are interested in applying Deep Learning (DL) to a wide variety of difficult problems.

I therefore have two questions:

  1. Practitioners - What do you find to be the main obstacles to applying DL 'out of the box' to your problem?
  2. Researchers - What techniques do you use (or have developed) that might help address practical issues? Are they within DL or do they offer an alternative approach?
  • 3
    $\begingroup$ If you have two questions, you should ask two questions. $\endgroup$ – bpachev Aug 4 '16 at 21:42
  • 1
    $\begingroup$ They are clearly inter-related. $\endgroup$ – NietzscheanAI Aug 4 '16 at 21:45

To summarize, There are two major issues in applied Deep Learning.

  • The first being that computationally , it's exhaustive. Normal CPU's require a lot of time to perform even the basic computation/training with Deep Learning. GPU's are thus recommended however, even they may not be enough in a lot of situations. Typical deep learning models don't support the theoretical time to be in Polynomials. However, if we look at the relatively simpler models in ML for the same tasks, too often we have mathematical guarantees that training time required for such simpler Algorithms is in Polynomials. This, for me, at least is probably the biggest difference.

    There're solutions to counter this issue, though. One main approach being is to optimize DL Algorithms to a number of iterations only (instead of looking at the global solutions in practice, just optimize the algorithm to a good local solution, whereas the criterion for "Good" is defined by the user).

  • Another Issue which may be a little bit controversial to young deep learning enthusiasts is that Deep Learning algorithms lack theoretical understanding and reasoning. Deep Neural Networks have been successfully used in a lot of situations including Hand writing recognition, Image processing, Self Driving Cars, Signal Processing, NLP and Biomedical Analysis. In some of these cases, they have even surpassed humans. However, that being said, they're not under any circumstance, theoretically as sound as most of Statistical Methods.

    I will not go into detail , rather I leave that up to you. There're pros and cons for every Algorithm/methodology and DL is not an exception. It's very useful as has been proven in a lot of situations and every young Data Scientist must learn at least the basics of DL. However, in the case of relatively simple problems, it's better to use famous Statistical methods as they have a lot of theoretical results/guarantees to support them. Furthermore, from learning point of view, it's always better to start with simple approaches and master them first.

  • $\begingroup$ By 'in polynomials' you mean 'in polynomial time', right? Have you got a reference to support that? $\endgroup$ – NietzscheanAI Mar 26 '18 at 17:56
  • $\begingroup$ Yes, that's exactly what I mean. Sure, It can be proved in a lot of situations...I will start with the Simplest possible example, Just training a Network with three Nodes, and two layers is NP-Complete problem as shown here.(citeseerx.ist.psu.edu/viewdoc/…) . Remember that this paper is very old, and now we have more ideas on how to improve in practice, with some heuristics, but still, theoretically, there're no improved results. $\endgroup$ – Sibghat Ullah Mar 27 '18 at 8:23
  • $\begingroup$ Other nice article on the same issue, which also describes some tricks to improve training time in practice. (pdfs.semanticscholar.org/9499/…) $\endgroup$ – Sibghat Ullah Mar 27 '18 at 8:28
  • $\begingroup$ Let's say, we want to predict the price for something. Simple Linear Regression with least square fit will have a Polynomial time, whereas solving the same issue with Neural Networks (even the simplest of them) will result in NP complete problem. This is a very big difference. Eventually, you have to carefully select an algorithm for a specific task. For example, Least Square fit has specific assumptions, which includes , "The ideal function which the algorithm is learning, can be learned as linear combination of features". If that assumption is not valid, so are results achieved. $\endgroup$ – Sibghat Ullah Mar 27 '18 at 8:36
  • $\begingroup$ Of course,simply because a problem (in this case, finding optimal weights) is NP-complete does not in itself mean that there aren't efficient practical methods for finding good weights... $\endgroup$ – NietzscheanAI Mar 27 '18 at 15:46

I have very little experience with ML/DL to call myself either practitioner, but here is my answer on the 1st question:

At its core DL solves well the task of classification. Not every practical problem can be rephrased in terms of classification. Classification domain needs to be known upfront. Although the classification can be applied to any type of data, it's necessary to train the NN with samples of the specific domain where it'll be applied. If the domain is switched at some point, while keeping the same model (NN structure), it'll have to be retrained with new samples. Furthermore, even the best classifiers have "gaps" - Adversarial Examples can be easily constructed from a training sample, such that changes are imperceptible to human, but are misclassified by the trained model.

  • 2
    $\begingroup$ 'Classification' can be considered a special case of 'regression', which is probably therefore a better characterization of DL. $\endgroup$ – NietzscheanAI Dec 14 '16 at 21:41

Question 2. I am researching whether Hyper dimensional computing is an alternative to Deep Learning. Hyper-D uses very long bit vectors (10,000 bits) to encode information. The vectors are random and as such they are approximately orthogonal. By grouping and averaging a collection of such vectors a "set" can be formed and later queried to see if an unknown vector belongs to the set. The set can be considered a concept or a generalize image, etc. Training is very fast as is recognition. What needs to be done is simulate the domains in which Deep Learning has been successful and compare Hyper-D with it.

  • $\begingroup$ Interesting. So how does this differ from Kanerva's 'Sparse Distributed Memory'? $\endgroup$ – NietzscheanAI Mar 26 '18 at 17:58
  • $\begingroup$ Both are developed by Pentti Kanerva. Look up Hyper dimensional computing to see the difference. Too long to answer here. $\endgroup$ – Douglas G Danforth Mar 28 '18 at 4:09

From a mathematics point of view one of the major issues in deep networks with several layers are vanishing or unstable gradients. Each additional hidden layer learns significantly slower, almost nullifying the benefit of the additional layer.

Modern deep learning approaches can improve this behavior, but in simple, old fashioned neural networks this is a well known issue. You can find a well written analysis here for deeper study.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.