4
$\begingroup$

I am working on an implementation of the back propagation algorithm. What I have implemented so far seems working but I can't be sure that the algorithm is well implemented, here is what I have noticed during training test of my network:

Specification of the implementation:

  • A data set containing almost 100000 raw containing (3 variable as input, the sinus of the sum of those three variables as expected output).
  • The network does have 7 layers, all the layers use the sigmoid activation function

When I run the back propagation training process:

  • The minimum of costs of the error is found at the fourth iteration (The minimum cost of error is 140, is it normal? I was expecting much less than that)
  • After the fourth iteration the costs of the error start increasing (I don't know if it is normal or not?)
Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ I would always output the average error, not just the average cost. For the average error of an approximation of sine 140 is way too high, obviously. I would also restrict the input to a certain range. It might be the case that the NN doesn't generalise to bigger inputs. Output your error for the training and the test set. And finally: Try a learning rate so small that almost nothing happens, then slowly increase. It might be that you are jumping around too much in the solution space. $\endgroup$ – BlindKungFuMaster Jan 26 '17 at 13:02
  • $\begingroup$ As you progress, you might want to try Residual connections to help with vanishing gradients. Search ResNets. Andrew Ng's Deep Learning ResNet video should be on Youtube. Check it out! $\endgroup$ – Ali_Ayub Jan 18 '19 at 5:17
4
$\begingroup$

Actually the implementation was correct,

The source of the problem that causes a big error and really slow learning was the architecture of the neural network it self, the ANN has 7 hidden layers which causes the vanishing gradient problem.

When I have decreased the ANN layers to 3 the cost of error was widely reduced besides of that the learning process was faster.

Another common solution is to use RELU or ELU or SELU in the hidden layers of the neural network instead of the sigmoid function

Improve this answer
$\endgroup$
3
  • $\begingroup$ Can you tell me how to deal with the vanishing gradient problem? $\endgroup$ – DuttaA Jan 26 '18 at 16:30
  • $\begingroup$ if you change the activation function to relu or elu that would fix the issue $\endgroup$ – Anas EL KORCHI Jan 27 '18 at 7:41
  • $\begingroup$ Yet another common solution is residual network, i.e., adding fixed weight connections skipping layers. I like it most as it works without modifying the structure / activation function. $\endgroup$ – maaartinus Mar 5 '18 at 7:52
3
$\begingroup$

I believe the best way to do this is using numerical gradient. To understand the concept, we need to look the definition of derivatives using limits:

enter image description here

It means that, when you don't know how to derive some formula (or you just don't want to), you can approximate it by computing the output for a small change in input, subtract from the original result (no change), and normalize by this change.

Example: We know the derivative of f(x) = x^2 is f'(x) = 2x. But, let suppose we don't and we are using x = 3 and h = 0.001 (it tends to zero in fact):

f(3 + 0.001) = (3 + 0.001)^2 = 9,006 (approximately)
f(3) = 3^2 = 9

Thus,

(9,006 - 9) / 0.001 = 6

It's is exactly to f'(3) = 2*3 = 6.

In practice, if you want to know if your backpropagation is correct,

  1. pass a single example (x1) through your network and computes the output (o1). Compute the gradient with respect to x1 (d1).
  2. Then, add a small value to the input (h), pass through the network again, and computes the new output (o2).
  3. Subtract o2 - o1 and divide by h. It must be close enough to d1.

That's it. It's called gradient checking. I hope it helps.

Improve this answer
$\endgroup$
2
$\begingroup$

One thing that you should know is that the sigmoid function limits the output to a value between 0 and 1, which means that using a lot of hidden layers will lead quickly to a Vanishing Gradient.

Try to use relu activation function, it has the property to output all the information it gets from the previous layer.

Improve this answer
$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.