Why does my neural network to solve the XOR problem always output 0.5?

Question

I'm trying to create a neural network to simulate an XOR gate.

Here's my dataset:

╔════════╦════════╗
║ x1, x2 ║ y1, y2 ║
╠════════╬════════╣
║  0, 0  ║  0, 1  ║
║  0, 1  ║  1, 0  ║
║  1, 0  ║  1, 0  ║
║  1, 1  ║  0, 1  ║
╚════════╩════════╝

And my neural network:

I use logistic loss to get the error between target $y_{k}$ and output $\hat{y}_{k}$:

$$ E(y_{k}, \hat{y}_{k}) = - y_{k} \cdot log(\hat{y}_{k}) + (1 - y_{k}) \cdot log(1 - \hat{y}_{k}) $$

And then use the chain rule to update the weights. For example weight $w_{3}$'s contribution to the error is:

$$ \sum_{k=1}^{2} \frac{\partial E(y_{k}, \hat{y}_{k})}{\partial w_{3}} = \sum_{k=1}^{2} \left(\frac{\partial E(y_{k}, \hat{y}_{k})}{\partial \hat{y}_{k}} \cdot \frac{\partial s_{k}}{\partial c_{1}}\right) \cdot \frac{\partial c_{1}}{\partial w_{3}} $$

Which in developed form is:

$$ \sum_{k=1}^{2} \frac{\partial E(y_{k}, \hat{y}_{k})}{\partial w_{3}} = \left( \left( - \frac{y_{1}}{\hat{y}_{1}} + \frac{1 - y_{1}}{1 - \hat{y}_{1}} \right) \cdot \hat{y}_{1} \cdot (1 - \hat{y}_{1}) + \left( - \frac{y_{2}}{\hat{y}_{2}} + \frac{1 - y_{2}}{1 - \hat{y}_{2}} \right) \cdot (- \hat{y}_{2}) \cdot\frac{c_{1}}{c_{1} + c_{2}} \right) \cdot s_{0} $$

My issue is that, after a couple epochs of training on the entire dataset, the network always outputs: $$ \hat{y}_{1} = \hat{y}_{2} = 0.5 $$

What am I doing wrong?

Answer 1

One neuron on its own can only solve linearly separable problems. You need a combination of neurons to solve non-linearly separable problems.

For the XOR case, you need at least 2 neurons at the first layer, and 1 neuron at the output layer to properly classify it.

Keep in mind that, sometimes, the 3-neurons network might get stuck in a local minimum as well. You will need some luck in the random initialization of weights. Using the right seed during the random initialization of weights can help converge, and some other seed will only result in a stuck network.