What is the proof behind the gradient of a curve being proportional to the distance between the two co-ordinates in the x-axis?

Question

In the [delta rule][1] the equation to adjust the weight with respect to error is

$$w_{(n+1)}=w_{(n)}-\alpha \times \frac{\partial E}{\partial w}$$

*where $\alpha$ is the learning rate and $E$ is the error.

The graph for $E$ vs $w$ would look like the one below with $E$ in the $y$ axis and $W$ in the $x$-axis

In other words, we can write

$$\alpha \times \frac{\partial E}{\partial w}=w_{(n)}-w_{(n+1)}$$

I want to know, what is the proof behind the gradient of a curve being equal/proportional to the distance between the two coordinates in the x-axis.

$\frac{\partial E}{\partial w}$ times step is a small shift on $f(w)$ not $w$. So, why does the difference between $W(n+1)$ and $W(n)$ be equal to $f(W)$?

I found a similar question, but the accepted answer doesn't have a proof.

Answer 1

Don’t think about it as the $w_{(n)}-w_{(n+1)}$ being proportional to something. Think about it this way:

I'm now at $w_{(n)}$. Where do I want to be at timestep, so that the error decreases? For that, I need to know how the error changes when I make small steps to the left or right of $w_{(n)}$.

If $E$ increases as I increase $w$ (that is, if $\frac{\partial E}{\partial w}>0$, then obviously, I would want to move a little bit to the left. In other words, $w_{n+1}<w_{n}$ or $w_{n+1}-w_{n}<0$.

On the other hand, if the derivative were negative, you know that you should move right to reduce the error a little bit, $w_{n+1}-w_{n}>0$. So, basically, your step should have the opposite sign of the derivative.

$$w_{n+1}-w_{n} \propto-\frac{\partial E}{\partial w}$$

$\alpha$, the learning rate, is just the constant of proportionality. Caution: think about small values for this rate, not big numbers. Taking a huge step can cause you to overshoot the minimum point.

Answer 2

$$let,\ x_{(n)} \ be\ a\ point\ on\ x-axis\ where\ f'( x) \ =\ 0\ ,\ and\ x_{(n\ +\ h)} \ is\ any\ other\ arbitary\ point \\ \therefore \ \ \frac{f'( x_{(n\ +\ h)})}{|\ f'( x_{(n\ +\ h)}) \ |} =\begin{cases} 1 & \mathrm{if,} \ h\ \ >0\\ 0 & \mathrm{if,} \ h\ =\ 0\\ -1 & \mathrm{if,} \ h\ < \ 0 \end{cases}\\similarly,\ \ \frac{x_{(n)} \ -\ x_{(n\ +\ h)} \ }{|\ x_{(n)} \ -\ x_{(n\ +\ h)} \ |} \ =\ \begin{cases} 1 & \mathrm{if,} \ h\ \ < 0\\ 0 & \mathrm{if,} \ h\ =\ 0\\ -1 & \mathrm{if,} \ h\ >\ 0 \end{cases}\\or,\ \frac{x_{(n)} \ -\ x_{(n\ +\ h)} \ }{|\ x_{(n)} \ -\ x_{(n\ +\ h)} \ |} \ =\ -\ \ \ \frac{f'( x_{(n\ +\ h)})}{|\ f'( x_{(n\ +\ h)}) \ |}\\ \therefore \ x_{(n)} \ =x_{(n\ +\ h)} \ -\ \eta \times f'( x_{(n\ +\ h)}) \ \ \ \ \ \ \left[ where\ \eta \ =\frac{|\ x_{(n)} \ -\ x_{(n\ +\ h)} \ |}{|f'( x_{(n\ +\ h)}) \ |} \ \right]$$