In the [delta rule][1] the equation to adjust the weight with respect to error is

$$w_{(n+1)}=w_{(n)}-\alpha \times \frac{\partial E}{\partial w}$$

*where $\alpha$ is the learning rate and $E$ is the error.

The graph for $E$ vs $w$ would look like the one below with $E$ in the $y$ axis and $W$ in the $x$-axis

enter image description here

In other words, we can write

$$\alpha \times \frac{\partial E}{\partial w}=w_{(n)}-w_{(n+1)}$$

I want to know, what is the proof behind the gradient of a curve being equal/proportional to the distance between the two coordinates in the x-axis.

$\frac{\partial E}{\partial w}$ times step is a small shift on $f(w)$ not $w$. So, why does the difference between $W(n+1)$ and $W(n)$ be equal to $f(W)$?

I found a similar question, but the accepted answer doesn't have a proof.


Don’t think about it as the $w_{(n)}-w_{(n+1)}$ being proportional to something. Think about it this way:

I'm now at $w_{(n)}$. Where do I want to be at timestep, so that the error decreases? For that, I need to know how the error changes when I make small steps to the left or right of $w_{(n)}$.

If $E$ increases as I increase $w$ (that is, if $\frac{\partial E}{\partial w}>0$, then obviously, I would want to move a little bit to the left. In other words, $w_{n+1}<w_{n}$ or $w_{n+1}-w_{n}<0$.

On the other hand, if the derivative were negative, you know that you should move right to reduce the error a little bit, $w_{n+1}-w_{n}>0$. So, basically, your step should have the opposite sign of the derivative.

$$w_{n+1}-w_{n} \propto-\frac{\partial E}{\partial w}$$

$\alpha$, the learning rate, is just the constant of proportionality. Caution: think about small values for this rate, not big numbers. Taking a huge step can cause you to overshoot the minimum point.


$$let,\ x_{(n)} \ be\ a\ point\ on\ x-axis\ where\ f'( x) \ =\ 0\ ,\ and\ x_{(n\ +\ h)} \ is\ any\ other\ arbitary\ point \\ \therefore \ \ \frac{f'( x_{(n\ +\ h)})}{|\ f'( x_{(n\ +\ h)}) \ |} =\begin{cases} 1 & \mathrm{if,} \ h\ \ >0\\ 0 & \mathrm{if,} \ h\ =\ 0\\ -1 & \mathrm{if,} \ h\ < \ 0 \end{cases}\\similarly,\ \ \frac{x_{(n)} \ -\ x_{(n\ +\ h)} \ }{|\ x_{(n)} \ -\ x_{(n\ +\ h)} \ |} \ =\ \begin{cases} 1 & \mathrm{if,} \ h\ \ < 0\\ 0 & \mathrm{if,} \ h\ =\ 0\\ -1 & \mathrm{if,} \ h\ >\ 0 \end{cases}\\or,\ \frac{x_{(n)} \ -\ x_{(n\ +\ h)} \ }{|\ x_{(n)} \ -\ x_{(n\ +\ h)} \ |} \ =\ -\ \ \ \frac{f'( x_{(n\ +\ h)})}{|\ f'( x_{(n\ +\ h)}) \ |}\\ \therefore \ x_{(n)} \ =x_{(n\ +\ h)} \ -\ \eta \times f'( x_{(n\ +\ h)}) \ \ \ \ \ \ \left[ where\ \eta \ =\frac{|\ x_{(n)} \ -\ x_{(n\ +\ h)} \ |}{|f'( x_{(n\ +\ h)}) \ |} \ \right]$$

  • $\begingroup$ I do not think you have proven anything here. In the equation second from bottom m and dy/dx cancel each out, right? What remains is x(n+h)-x(n)=h, which is kind of trivial, i.e. follows from notations you’ve chosen in (I) and (II). $\endgroup$
    – aivanov
    Mar 21 '18 at 21:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.