Just for fun, I am trying to develop a neural network.
Now, for backpropagation I saw two techniques.
The first one is used here and in many other places too.
What it does is:
- It computes the error for each output neuron.
- It backpropagates it into the network (calculating an error for each inner neuron).
- It updates the weights with the formula: (where is the change in weight, the learning speed, the error of the neuron receiving the input from the synapse and being the output sent on the synapse).
- It repeats for each entry of the dataset, as many times as required.
However, the neural network proposed in this tutorial (also available on GitHub) uses a different technique:
- It uses an error function (the other method does have an error function, but it does not use it for training).
- It has another function which can compute the final error starting from the weights.
- It minimizes that function (through gradient descent).
Now, which method should be used?
I think the first one is the most used one (because I saw different examples using it), but does it work as well?
In particular, I don't know:
- Isn't it more subject to local minimums (since it doesn't use quadratic functions)?
- Since the variation of each weight is influenced by the output value of its output neuron, don't entries of the dataset which just happen to produce higher values in the neurons (not just the output ones) influence the weights more than other entries?
Now, I do prefer the first technique, because I find it simpler to implement and easier to think about.
Though, if it does have the problems I mentioned (which I hope it doesn't), is there any actual reason to use it over the second method?