I've been trying to implement a Multilayer Perceptron Network using java language with the ultimate goal of creating and teaching a neural network to recognize handwritten digits. Pretty simple and straight forward. This is the link to the repository of the project. I tried to write everything as simple as I could. When the class was ready to test, I provided the network with suitable inputs (around 3000 training samples) but the network doesn't seem to converge to a minimum point. Instead for every input it just gives a vector output of 1s." I think that the problem is with the implementation of the backpropagation algorithm, though I'm not 100% sure of it. I use a book called Fundamentals of Computational Intelligence by Keller, Liu, and Fogel as the source for all the maths behind ANN. I try to explain my code and my understanding of the math as vividly as I can.

The Backpropagation code:

enter image description here

Sorry that I didn't type the code. I needed to refer to some specific lines and I couldn't if I wrote the code down.

So the algorithm begins from line 392 with a loop that calculates the error for each neuron. this.weigths.length is the number of network's layers. So I'm iterating in layers with the first loop and as you can see I'm starting from the output layer and work my way back to input layer.
In line 394 I'm referring that if the program is iterating on the neurons of the output layer, the error of each node is calculatable by subtracting the desired output of the network from the actual output, After that, when the iteration of layers gets to the hidden layers of the network, the error for each neuron is calculatable by the following rule: $$\displaystyle\sum_{l}^{\#(l)} e_l \phi_l'(v_{l,k})w_{lj}$$ In which $l$ represents a node in the next layer of the network, $k$ represents the input data (for a given input data $k$) and $j$ represents the current node that we are calculating the error for. Likewise $\phi(*)$ is the activation function so $\phi'(*)$ is its derevitive, $e_l$ is the error of node $l$, $w_{lj}$ is the weight of the connection which connects node $j$ to node $l$ and finally $v_{l,k}$ is the signal of node $l$ given a sample $k$ in the input.
I'm saving all of these in a double array called errors.
Using this error array, from line 414 forward, the program is calculating $\Delta w_{ij}$ which is a correction for the weight of the connection that connects node $i$ to $j$. $$\Delta w_{ij} = \alpha e_{(j,k)} \phi'_j(v_{(j,k)}) y_{(i,k)} $$ note that $y_{(i,k)}$ represents the signal output of neuron $i$ for a given sample $k$
In the program outputs is $y$, fields is $v$, errors is $e$, activation_function_derivative is $\phi'$ , learningRate is $\alpha$ and weights are $w$
I personally found these two pictures verry helpful enter image description here enter image description here

Now with all of this said, I don't know where I made a mistake in understanding the MLP's math that my program is not working. It'll be of much help if you could help me a little bit. I've been going blind for 3 days staring at monitors and textbooks.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.