# Neural Network learning XOR. I collected Data on my networks convergence. Is this expected behavior?

I build a neural network from scratch to get a better understanding of the fundamentals of machine learning.

The network contains a bias for each neuron and calculates the final error via the mean squared error formular. Weights are updated through classic backpropagation. Training does not happen in batches.

The first step to test and try out the network was training XOR. I used 2 inputs, 2 nodes in a single hidden layer, and one output.

The networks is able to learn XOR sometimes. I checked the code for bugs several times and even calculated one cycle (forward & backward propagation) by hand to compare results (they were identical).

Then this answer gave me the idea to compare my convergences to other networks.

What I am unsure about is the behavior of the neural networks with regards to the convergence. Since I am new to the whole topic of machine learning I would like to get some feedback on the following data I collected with the network. I have 2 questions that can be found below.

I examined the convergence with different activation functions:

• sigmoid
• ReLU
• leaky ReLU (max(0.01x, x))
• leaky ReLU (max(0.001x, x))

and different learning rates:

• 0.5
• 0.3
• 0.1
• 0.01

and different intervals to choose the random weights from

• [0, 1]
• [-0.5, 0.5]
• [0, 0.5]
• [0.5, 1]

The following plots show the collected data. It shows the rate of non-convergence with different parameters (30 samples). Each plot represents one activation function.

Please note that the second weight interval is [-0.5, 0.5] and not [-.05, 0.5] as wrongly shown in the axis label:

every combination of parameters never converges with the sigmoid function...

even the 'best' learning rate only converges 93% of the time...

even the 'best' learning rate only converges 93% of the time...

even the 'best' learning rate only converges 93% of the time...

My first question is: Is it expected behavior, that there is never a 0% non-convergence rate? Could this be an indication that my network has a bug?

Afterwards I collected data on the loss function behavior during training for those 4 combinations of parameters, that resulted in the lowest non-convergence rate of 7%. The loss over time is shown here:

This data brings me to my second question: Why is the loss decreasing so stepwise and not decreasing in a smoother manner? Could this be an indication that my network has a bug?

If anybody wants to look at the code, it can be found at this repository (including all the collected data in the README). The networks code is in cnn.ts, it is written in typescript.

• Anybody? Would love to discuss the results! :) Apr 2 at 6:49