0
$\begingroup$

I'm trying to create a neural network to simulate a XOR gate.

Here's my dataset:

╔════════╦════════╗
║ x1, x2 ║ y1, y2 ║
╠════════╬════════╣
║  0, 0  ║  0, 1  ║
║  0, 1  ║  1, 0  ║
║  1, 0  ║  1, 0  ║
║  1, 1  ║  0, 1  ║
╚════════╩════════╝

And my neural network:

enter image description here

I use logistic loss to get the error between target $y_{k}$ and output $\hat{y}_{k}$:

$$ E(y_{k}, \hat{y}_{k}) = - y_{k} \cdot log(\hat{y}_{k}) + (1 - y_{k}) \cdot log(1 - \hat{y}_{k}) $$

And then use the chain rule to update the weights. For example weight $w_{3}$'s contribution to the error is:

$$ \sum_{k=1}^{2} \frac{\partial E(y_{k}, \hat{y}_{k})}{\partial w_{3}} = \sum_{k=1}^{2} \left(\frac{\partial E(y_{k}, \hat{y}_{k})}{\partial \hat{y}_{k}} \cdot \frac{\partial s_{k}}{\partial c_{1}}\right) \cdot \frac{\partial c_{1}}{\partial w_{3}} $$

Which in developed form is:

$$ \sum_{k=1}^{2} \frac{\partial E(y_{k}, \hat{y}_{k})}{\partial w_{3}} = \left( \left( - \frac{y_{1}}{\hat{y}_{1}} + \frac{1 - y_{1}}{1 - \hat{y}_{1}} \right) \cdot \hat{y}_{1} \cdot (1 - \hat{y}_{1}) + \left( - \frac{y_{2}}{\hat{y}_{2}} + \frac{1 - y_{2}}{1 - \hat{y}_{2}} \right) \cdot (- \hat{y}_{2}) \cdot\frac{c_{1}}{c_{1} + c_{2}} \right) \cdot s_{0} $$

My issue is that after a couple epochs of training on the entire dataset, the network always outputs: $$ \hat{y}_{1} = \hat{y}_{2} = 0.5 $$

What am I doing wrong?

$\endgroup$
0
$\begingroup$

One Neuron on its own can only solve linearly separable problems.
You need a combination of Neurons to solve non-linearly separable problems.

For the XOR case, you need at least 2 Neuron at the first layer, and 1 Neuron at the Output layer to properly classify it.
Keep in mind sometimes the 3 Neuron network might get stuck in a local minima as well, you will need some luck in the random initialization of weights.
Using the right seed during the random initialization of weights can help converge, and some other seed will only result in a stuck network.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.