There are a lot of papers that show that neural networks can approximate a wide variety of functions. However, I can't find papers that show the limitations of NNs.

What are the limitations of neural networks? Which functions can't neural networks learn efficiently (or using gradient-descent)?

I am looking also for links to papers that describe these limitations.

  • $\begingroup$ I want clarify. For example lets consider feedforfard NN. Yes it can represent anything but for some tasks it could demand too many data or too many computational power or incredible count of neurons and layers. for example X1*X2. It can be represented and this function is very simple and we expect that our cool approximator can do it easy but it actualy can only inefficient proximate it by creation a banch of hypeplane spliting space. $\endgroup$ – user2674414 Mar 7 '18 at 12:27
  • $\begingroup$ The fact that you cannot find a paper about limitations... I don't know how that would even happen. It seems almost impossible to get something setup, and not read the warning page. $\endgroup$ – FreezePhoenix Apr 11 '18 at 13:46

One of the important qualifications of the Universal approximation theorem is that the neural network approximation may be computationally infeasible.

"A feedforward network with a single layer is sufficient to represent any function, but the layer may be infeasibly large and may fail to learn and generalize correctly." - Ian Goodfellow, DLB

I can't think of any function that I would definitively declare as unlearnable, but neural networks have many problems. Consider adversarial examples and adversarial patches, which highlight the poor generalization going on under the hood of recent advances in computer vision.

Neural Networks are also inherently limited by the innate priors baked into their architecture and the sample density of their training data. Check out this recent discussion at Stanford's AI Salon between Yann LeCun and Christopher Manning on innate priors if that is the kind of limitation you are talking about.

  • $\begingroup$ For example X1 * X2, this function can be approximated by feedforward NN but can't be efficiently represented by splitting space with linear hyperplanes, also NN poor with fitures combinations like x1*x2, x1/x2, 1/x1 and so on, because neuron creates only linear hyperplane and it isn't efficient. $\endgroup$ – user2674414 Mar 7 '18 at 12:15

This answer depends very much so on the type of neural network and algorithm used for training.

If you are using gradient descent on a neural network of one input layer, one output layer, and no hidden layers there are many functions that you can't learn. One simple one is the XOR function. Due to the fact that XOR is not linearly separable, it can not be represented by a neural network with no hidden layers.

If you are using NEAT to build recurrent neural networks then all functions(**) can be represented given enough time and data. This is due in part to the fact that recurrent neural networks are Turing Complete.

One of the biggest causes for limitations when using neural networks is based on the difficulty of interpretation as to what the network is doing. The network is gradually building up an understanding of the function as it goes from the input layer to the output layer, but it is very difficult for us to understand this building up process and interpret what the neural network is attempting to do. This makes it very hard if not impossible to manually tweak your neural network in a meaningful way.

Another limitation is the need for training (in large amounts) in order to have a meaningful representation of your data. Neural networks have a tendency to need large amounts of data before converging to a meaningful hypothesis space. This has resulted in clever algorithms to generate training data without needing human interaction, such as Generative Adversarial Networks, but the underlying problem remains.

** Not all functions can be computed by neural networks, however all computable functions can be. An example of an uncomputable function is the mapping of all programs from the program to whether or not this program will halt (The Halting Problem).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.