# Why is the error curve of a neural network trained with MSE to output $\frac{3 I_1 + 5 I_2}{2}$ given inputs $I_1$ and $I_2$ oscillating weirdly?

I just "finished" my first AI program. I programmed in Excel VBA, and I think it works well. I was checking every formula and the whole algorithm several times to make sure every formula is correct.

This picture is basically when the ANN is learning. The loss function is the mean squared error

$$E = \sum_{i=0}^n \left(\hat{y}_i - y_i \right) ^2$$

If I change the learning speed $$w^* = w - \eta * \frac{\delta E}{\delta weight},$$ then, at some point, the cost function behaves really weirdly, and eventually explodes in either negative or positive direction.

Do you guys know why that might happen? I was expecting the error curve to go closer to 0 at every iteration. I changed the learning rate from the first picture to the second one from 0.0005 to 0.001.

Why should the derivative suddenly go in the other direction?

I just have 2 real-valued inputs $$I_1$$ and $$I_2$$ and one output being just $$\frac{3 I_1 + 5 I_2}{2}$$, which should not be a problem for an ANN with 3 layers, with 8 neurons each. Right? Every neuron has leaky ReLU activation function, so at all times there are fix values for the derivation.

### Edit

I found out that this is an issue due to high weights. So, I made the check whether $$\frac{\delta E}{\delta weight} > 1$$. If that is the case because of big weights $$\frac{\delta E}{\delta weight}$$ is set to 1 (vice versa for -1).

The result is following picture:

I don't know where that does come from. At least it did not explode. Do you have some problem like this or some guess on what it may be?

• I edited your post to put what I think is your main question in the title. Make sure that your post is still consistent with what you were asking.
– nbro
Mar 9 at 10:30