solving xor function using a neural network with no hidden layers

Question

xor is a non-linear dataset. It cannot be solved with any number of perceptron based neural network but when the perceptions are applied the sigmoid activation function, we can solve the xor dataset.

But I came across a source where the following statement is stated as False

A two layer (one input layer; one output layer; no hidden layer) neural network can represent the XOR function.

However, I have trained a model with no hidden layers, gives the following result:

[INFO] data=[0 0],ground-truth=0, pred=0.5161, step=1
[INFO] data=[0 1],ground-truth=1, pred=0.5000, step=1
[INFO] data=[1 0],ground-truth=1, pred=0.4839, step=0
[INFO] data=[1 1],ground-truth=0, pred=0.4678, step=0

So, in if I apply a softmax classifier, I can separate the xor dataset with a nn without any hidden layer. This makes the statement incorrect.

Is it true that we cannot separate a non linear dataset without any hidden layers in a neural network? If yes, where am I wrong in my reasoning from the training of the nn I have done above

Answer 1

Softmax is a probability distribution you use when you want probability for all multiple classes you are predicting which are not independent, ie, exp(xi)/sum(exp(xj) for j in all x), where xi is the score of one neuron, so softmax is good if you have more than 1 neurons, but for just 1 neuron(in this case), the output of softmax will be 1, always (exp(xi)/exp(xi)).

Now lets visualize your dataset, ie plot the points of XOR as a function of X,Y on coordinate space. , [src: https://www.researchgate.net/figure/The-exclusive-or-XOR-function-is-a-nonlinear-function-that-returns-0-when-its-two_fig4_322048911]

Now, whatever you do, you cannot separate the points with a single wx+b straight line, and after passing it to sigmoid, you just squeeze the values for those, ie, a clipped wx+b.

Now, it is still a linear equation. Now when you add another layer, a hidden one, you can operate again on the 1st output, which if you squeeze between 0 and 1 or use something like relu activation, will produce some non linearity, otherwise it will just be (w2(w1*x + b1)+b2, which again is a linear equation not able to separate the classes 0 and 1.

So after adding a hidden layer and passing it through activation function, you get non linear output, example, with sigmoid you get w2*(1/+exp(w2*x+b1))+b2 which has a fairly non linear term of x, and after passing it through sigmoid again, you are able to squeeze the outputs(otherwise they can be higher than 1 or lower than 0,but XOR outputs are binary) after fitting the curve so that both classes are separate.

Now, food for thought, what will happen if you use 2 output neurons for XOR (represent 0 as [1,0] and 1 as [0,1])?