0
$\begingroup$

I build a neural network from scratch to get a better understanding of the fundamentals of machine learning.

The network contains a bias for each neuron and calculates the final error via the mean squared error formular. Weights are updated through classic backpropagation. Training does not happen in batches.

The first step to test and try out the network was training XOR. I used 2 inputs, 2 nodes in a single hidden layer, and one output.

The networks is able to learn XOR sometimes. I checked the code for bugs several times and even calculated one cycle (forward & backward propagation) by hand to compare results (they were identical).

Then this answer gave me the idea to compare my convergences to other networks.

What I am unsure about is the behavior of the neural networks with regards to the convergence. Since I am new to the whole topic of machine learning I would like to get some feedback on the following data I collected with the network. I have 2 questions that can be found below.

I examined the convergence with different activation functions:

  • sigmoid
  • ReLU
  • leaky ReLU (max(0.01x, x))
  • leaky ReLU (max(0.001x, x))

and different learning rates:

  • 0.5
  • 0.3
  • 0.1
  • 0.01

and different intervals to choose the random weights from

  • [0, 1]
  • [-0.5, 0.5]
  • [0, 0.5]
  • [0.5, 1]

The following plots show the collected data. It shows the rate of non-convergence with different parameters (30 samples). Each plot represents one activation function.

Please note that the second weight interval is [-0.5, 0.5] and not [-.05, 0.5] as wrongly shown in the axis label:

non-convergence rate with sigmoid activation every combination of parameters never converges with the sigmoid function...

non-convergence rate with ReLU activation even the 'best' learning rate only converges 93% of the time...

non-convergence rate with leaky ReLU (max(0.01x,x)) activation even the 'best' learning rate only converges 93% of the time...

non-convergence rate with leaky ReLU (max(0.001x,x)) activation even the 'best' learning rate only converges 93% of the time...

My first question is: Is it expected behavior, that there is never a 0% non-convergence rate? Could this be an indication that my network has a bug?

Afterwards I collected data on the loss function behavior during training for those 4 combinations of parameters, that resulted in the lowest non-convergence rate of 7%. The loss over time is shown here:

ReLU, weights in [0, 1], learning rate 0.01

ReLU, weights in [0.5, 1], learning rate 0.01

leaky ReLU with 0.01x, weights in [0.5, 1], learning rate 0.01

leaky ReLU with 0.001x, weights in [0, 1], learning rate 0.01

This data brings me to my second question: Why is the loss decreasing so stepwise and not decreasing in a smoother manner? Could this be an indication that my network has a bug?

If anybody wants to look at the code, it can be found at this repository (including all the collected data in the README). The networks code is in cnn.ts, it is written in typescript.

$\endgroup$
1
  • $\begingroup$ Anybody? Would love to discuss the results! :) $\endgroup$
    – felixmp
    Apr 2 at 6:49

0

You must log in to answer this question.