xor is a non-linear dataset. It cannot be solved with any number of perceptron based neural network but when the perceptions are applied the sigmoid activation function, we can solve the xor dataset.

But I came across a source where the following statement is stated as False

A two layer (one input layer; one output layer; no hidden layer) neural network can represent the XOR function.

However, I have trained a model with no hidden layers, gives the following result:

[INFO] data=[0 0],ground-truth=0, pred=0.5161, step=1
[INFO] data=[0 1],ground-truth=1, pred=0.5000, step=1
[INFO] data=[1 0],ground-truth=1, pred=0.4839, step=0
[INFO] data=[1 1],ground-truth=0, pred=0.4678, step=0

So, in if I apply a softmax classifier, I can separate the xor dataset with a nn without any hidden layer. This makes the statement incorrect.

Is it true that we cannot separate a non linear dataset without any hidden layers in a neural network? If yes, where am I wrong in my reasoning from the training of the nn I have done above

  • $\begingroup$ What do you think softmax does? what change will it make if you replace a sigmoid with a softmax when the output layer is only one neuron? $\endgroup$ – ctlr May 4 '20 at 13:00
  • $\begingroup$ I meant placing the softmax after the last layer, but now realize that it would no longer be a nn with no hidden lalyers $\endgroup$ – Hrushi May 4 '20 at 13:06
  • $\begingroup$ last layer has weights, you just replace a sigmoid with a softmax $\endgroup$ – ctlr May 4 '20 at 13:07
  • $\begingroup$ But adding a softmax or a sigmoid will make the output of the network nonlinear right (because sigmoid or softmax are nonlinear functions), and the xor is a non linear dataset too. Is there any proof/justification that we cannot separate $\endgroup$ – Hrushi May 4 '20 at 13:14

Softmax is a probability distribution you use when you want probability for all multiple classes you are predicting which are not independent, ie, exp(xi)/sum(exp(xj) for j in all x), where xi is the score of one neuron, so softmax is good if you have more than 1 neurons, but for just 1 neuron(in this case), the output of softmax will be 1, always (exp(xi)/exp(xi)).

Now lets visualize your dataset, ie plot the points of XOR as a function of X,Y on coordinate space. XOR plot image, [src: https://www.researchgate.net/figure/The-exclusive-or-XOR-function-is-a-nonlinear-function-that-returns-0-when-its-two_fig4_322048911]

Now, whatever you do, you cannot separate the points with a single wx+b straight line, and after passing it to sigmoid, you just squeeze the values for those, ie, a clipped wx+b.

Now, it is still a linear equation. Now when you add another layer, a hidden one, you can operate again on the 1st output, which if you squeeze between 0 and 1 or use something like relu activation, will produce some non linearity, otherwise it will just be (w2(w1*x + b1)+b2, which again is a linear equation not able to separate the classes 0 and 1.

So after adding a hidden layer and passing it through activation function, you get non linear output, example, with sigmoid you get w2*(1/+exp(w2*x+b1))+b2 which has a fairly non linear term of x, and after passing it through sigmoid again, you are able to squeeze the outputs(otherwise they can be higher than 1 or lower than 0,but XOR outputs are binary) after fitting the curve so that both classes are separate.

Now, food for thought, what will happen if you use 2 output neurons for XOR (represent 0 as [1,0] and 1 as [0,1])?

  • 1
    $\begingroup$ Thank you for the answer and a wonderful question. In my opinion, using 2 output neurons will not be sufficient because each of the output from neurons is still just 'clipped' from a linear function. But the xor data(even though it's representation is by a matrix now) is still non linear, we just increased it's space representation from 1d to 2d. It would be great, if you can reason your question $\endgroup$ – Hrushi May 4 '20 at 16:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.