# XOR Neural Network gets stuck in training

I'm trying to create a neural network to simulate a XOR gate.

Here's my dataset:

╔════════╦════════╗
║ x1, x2 ║ y1, y2 ║
╠════════╬════════╣
║  0, 0  ║  0, 1  ║
║  0, 1  ║  1, 0  ║
║  1, 0  ║  1, 0  ║
║  1, 1  ║  0, 1  ║
╚════════╩════════╝


And my neural network:

I use logistic loss to get the error between target $$y_{k}$$ and output $$\hat{y}_{k}$$:

$$E(y_{k}, \hat{y}_{k}) = - y_{k} \cdot log(\hat{y}_{k}) + (1 - y_{k}) \cdot log(1 - \hat{y}_{k})$$

And then use the chain rule to update the weights. For example weight $$w_{3}$$'s contribution to the error is:

$$\sum_{k=1}^{2} \frac{\partial E(y_{k}, \hat{y}_{k})}{\partial w_{3}} = \sum_{k=1}^{2} \left(\frac{\partial E(y_{k}, \hat{y}_{k})}{\partial \hat{y}_{k}} \cdot \frac{\partial s_{k}}{\partial c_{1}}\right) \cdot \frac{\partial c_{1}}{\partial w_{3}}$$

Which in developed form is:

$$\sum_{k=1}^{2} \frac{\partial E(y_{k}, \hat{y}_{k})}{\partial w_{3}} = \left( \left( - \frac{y_{1}}{\hat{y}_{1}} + \frac{1 - y_{1}}{1 - \hat{y}_{1}} \right) \cdot \hat{y}_{1} \cdot (1 - \hat{y}_{1}) + \left( - \frac{y_{2}}{\hat{y}_{2}} + \frac{1 - y_{2}}{1 - \hat{y}_{2}} \right) \cdot (- \hat{y}_{2}) \cdot\frac{c_{1}}{c_{1} + c_{2}} \right) \cdot s_{0}$$

My issue is that after a couple epochs of training on the entire dataset, the network always outputs: $$\hat{y}_{1} = \hat{y}_{2} = 0.5$$

What am I doing wrong?

## 1 Answer

One Neuron on its own can only solve linearly separable problems.
You need a combination of Neurons to solve non-linearly separable problems.

For the XOR case, you need at least 2 Neuron at the first layer, and 1 Neuron at the Output layer to properly classify it.
Keep in mind sometimes the 3 Neuron network might get stuck in a local minima as well, you will need some luck in the random initialization of weights.
Using the right seed during the random initialization of weights can help converge, and some other seed will only result in a stuck network.