Why is non-linearity desirable in a neural network?

I couldn't find satisfactory answers to this question on the web. I typically get answers like "real-world problems require non-linear solutions, which are not trivial. So, we use non-linear activation functions for non-linearity".


Consider what happens if you intend to train a linear classifier on replicating something trivial as the XOR function. If you program/train the classifier (of arbitrary size) such that it outputs XOR condition is met whenever feature a or feature b are present, then the linear classifier will also (incorrectly) output XOR condition is met whenever both features together are present. That is because linear classifiers simply sum up contributions of all features and work with the total weighted inputs they receive. For our example, that means that when the weighted contribution of either feature is sufficient already to trigger the classifier to output XOR condition is met, then obviously also the summed contributions of both features are sufficient to trigger the same response.

To get a classifier which is capable of outputting XOR condition is met if and only if the summed contributions of all input features are above a lower threshold and below an upper threshold, commonly non-linearities are introduced. You could of course also try to employ a quadratic function to solve the two-feature problem, but as soon as the number of variables/features increases again, you run into the same problem again, only in higher dimensions. Therefore, the most general approach to solving this problem of learning non-linear functions, like the XOR, is by setting up large models with enough capacity to learn a given task and equipping them with non-linearities. That simplifies training, since it allows for using stochastic gradient descent for training the system/classifier, preventing one from having to solve Higher-Degree polynomial equations analytically (which can get computationally quite expensive quite quickly) to solve some task.

In case you are interested, here's one paper analyzing and dealing with the XOR problem (as one concrete instance of a problem where purely linear models fail to solve some task).


You can consider a layer in a network as a function y = f(x), where x is the input to some layer f and y is the output of the layer. As you propagate x, being the network's input, through the network, you get something like y = p(t(h(g(f(x))))), where f is the input layer and p constitutes the output layer, i.e. a set of weights, by which the input to that respective layer gets multiplied. If h, for example, is some non-linear activation function, like ReLU or sigmoid, then y, being the network's output, is a non-linear function of input x.

  • $\begingroup$ I think that a simple diagram that illustrates that a straight-line cannot solve the XOR problem and that you need a non-linear function to solve would give the intuition. I suppose that there are diagrams on the web that show this issue well. $\endgroup$ – nbro Jun 25 '20 at 13:32
  • $\begingroup$ what types of diagrams are you talking about? $\endgroup$ – Alankar Shukla Jun 26 '20 at 15:31
  • $\begingroup$ @AlankarShukla For example, these diagrams show that a straight-line cannot solve the XOR problem. By the way, next time, tag me with @nbro, otherwise, I may not see your message. $\endgroup$ – nbro Jun 27 '20 at 22:36
  • $\begingroup$ @nbro i wanna know what is a non linear data.how it can introduced in neural networks,how the neural network learns the non linear data for better prediction. $\endgroup$ – Alankar Shukla Jun 30 '20 at 3:46
  • $\begingroup$ @nbro i can't get the intuition behind it.i want to know what is a non linear data,how it is introduced in neural networks,and how neural networks learn these data for predicting. $\endgroup$ – Alankar Shukla Jun 30 '20 at 3:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.