Are these two versions of back-propagation equivalent?

Question

Just for fun, I am trying to develop a neural network.

Now, for backpropagation I saw two techniques.

The first one is used here and in many other places too.

What it does is:

• It computes the error for each output neuron.
• It backpropagates it into the network (calculating an error for each inner neuron).

• It updates the weights with the formula: $\Delta w_{l, m, n}=k \cdot E_{l+1, n} \cdot N_{l, m}$, where

  • $\Delta w_{l, m, n}$ is the change in weight,
  • $k$ is the learning speed,
  • $E_{l+1, n}$ is the error of the neuron receiving the input from the synapse, and
  • $ N_{l, m}$ is the output sent on the synapse.

• It repeats for each entry of the dataset, as many times as required.

However, the neural network proposed in this tutorial (also available on GitHub) uses a different technique:

• It uses an error function (the other method does have an error function, but it does not use it for training).
• It has another function which can compute the final error starting from the weights.
• It minimizes that function (through gradient descent).

Now, which method should be used?

I think the first one is the most used one (because I saw different examples using it), but does it work as well?

In particular, I don't know:

• Isn't it more subject to local minimums (since it doesn't use quadratic functions)?
• Since the variation of each weight is influenced by the output value of its output neuron, don't entries of the dataset which just happen to produce higher values in the neurons (not just the output ones) influence the weights more than other entries?

Now, I do prefer the first technique, because I find it simpler to implement and easier to think about.

Though, if it does have the problems I mentioned (which I hope it doesn't), is there any actual reason to use it over the second method?

Answer 1

The two examples present essentially the same operation:

• In both cases, the network is trained with gradient descent using the backpropagated squared error computed at the output.

• Both examples use the logistic function for node activation (the derivative of the logistic function $s$ is $s(1 - s)$. This derivative is obviously very easy to compute, and this is part of the reason why it was so widely used (these days the ReLU activation function is more popular, especially with convolutional networks).

• The first method also uses momentum.

The main difference I can see is that in the first case backpropagation is iterative while in the second example it is performed in batch mode.

The last video in the series by Welch Labs introduces a quasi-Newtonian method which offers the advantage of finding the minimum of the cost function by computing the Hessian (matrix of second-order derivatives of the error with respect to the weights). However, this feels like comparing apples and oranges - the vanilla gradient descent does not use second-order information.