There are a lot of papers that show that neural networks can approximate a wide variety of functions. However, I can't find papers that show the limitations of NNs.

What are the limitations of neural networks? Which functions can't neural networks learn efficiently (or using gradient-descent)?

I am looking also for links to papers that describe these limitations.

  • $\begingroup$ I want clarify. For example lets consider feedforfard NN. Yes it can represent anything but for some tasks it could demand too many data or too many computational power or incredible count of neurons and layers. for example X1*X2. It can be represented and this function is very simple and we expect that our cool approximator can do it easy but it actualy can only inefficient proximate it by creation a banch of hypeplane spliting space. $\endgroup$ – user2674414 Mar 7 '18 at 12:27
  • $\begingroup$ The fact that you cannot find a paper about limitations... I don't know how that would even happen. It seems almost impossible to get something setup, and not read the warning page. $\endgroup$ – FreezePhoenix Apr 11 '18 at 13:46

One of the important qualifications of the Universal approximation theorem is that the neural network approximation may be computationally infeasible.

"A feedforward network with a single layer is sufficient to represent any function, but the layer may be infeasibly large and may fail to learn and generalize correctly." - Ian Goodfellow, DLB

I can't think of any function that I would definitively declare as unlearnable, but neural networks have many problems. Consider adversarial examples and adversarial patches, which highlight the poor generalization going on under the hood of recent advances in computer vision.

Neural Networks are also inherently limited by the innate priors baked into their architecture and the sample density of their training data. Check out this recent discussion at Stanford's AI Salon between Yann LeCun and Christopher Manning on innate priors if that is the kind of limitation you are talking about.

  • $\begingroup$ For example X1 * X2, this function can be approximated by feedforward NN but can't be efficiently represented by splitting space with linear hyperplanes, also NN poor with fitures combinations like x1*x2, x1/x2, 1/x1 and so on, because neuron creates only linear hyperplane and it isn't efficient. $\endgroup$ – user2674414 Mar 7 '18 at 12:15

This answer depends very much so on the type of neural network and algorithm used for training.

If you are using gradient descent on a neural network of one input layer, one output layer, and no hidden layers there are many functions that you can't learn. One simple one is the XOR function. Due to the fact that XOR is not linearly separable, it can not be represented by a neural network with no hidden layers.

If you are using NEAT to build recurrent neural networks then all functions(**) can be represented given enough time and data. This is due in part to the fact that recurrent neural networks are Turing Complete.

One of the biggest causes for limitations when using neural networks is based on the difficulty of interpretation as to what the network is doing. The network is gradually building up an understanding of the function as it goes from the input layer to the output layer, but it is very difficult for us to understand this building up process and interpret what the neural network is attempting to do. This makes it very hard if not impossible to manually tweak your neural network in a meaningful way.

Another limitation is the need for training (in large amounts) in order to have a meaningful representation of your data. Neural networks have a tendency to need large amounts of data before converging to a meaningful hypothesis space. This has resulted in clever algorithms to generate training data without needing human interaction, such as Generative Adversarial Networks, but the underlying problem remains.

** Not all functions can be computed by neural networks, however all computable functions can be. An example of an uncomputable function is the mapping of all programs from the program to whether or not this program will halt (The Halting Problem).


To answer the question, I want to go a small sidestep to the first neural network ever invented, the McCulloch-Pitts Neurons from 1943. From today's perspective it may be surprising, but these primitive neurons are Turing complete, that means they can compute any algorithm like a “Neural Turing Machine (NTM)”. The reason is, because the McCulloch-Pitts neurons are implementations of logicgates (and, or, not) which can be connected to any function, including a counter, a for loop and a prime-number-checker.

Neural networks have no limitations from a computing perspective, because today's neural networks are more powerful than the former McCulloch Pitts neurons. The limitation is only the gradient descent algorithm for finding the weights. The learning algorithm is not very efficient and there is no better alternative available. In the error map, the problem must be reduced to zero-error, that means the neural network outputs the right value. Gradient descent is a heuristic to follow a certain path in the maze in the hope, that the door out is at the end. In many cases, gradient descent finds only an impasse. Perhaps an example:

A neural network should print out, if the input number is a prime number or not. With gradient descent the error rate can reduced to the value “5.4”. In theory, the neural network is able to find the prime-number algorithm, that means to reduce the error rate down to 0.0. But the gradient descent algorithm doesn't know how to reduce the error-rate from “5.4” to “5.3” and further. He is iin a local minimum. The reason is, that the error map is under the fog-of-war, to uncover the information CPU-intensive trial&error search has to be done.

Bypassing the problem is easy: The programmer has to implement the problem-solving algorithm by hand in a normal programming language, and use a neural network only for finding a parameter in his algorithm.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.