0
$\begingroup$

In the delta rule the equation to adjust the weight with respect to error is :-

enter image description here

where enter image description here is the Learning Rate and E is the Error

The graph for E vs w would look like the one below with E in the y axis and W in the x axis

enter image description here

In other words we can write

enter image description here

I want to know, what is the proof behind the gradient of a curve being equal/proportional to the distance between the two co-ordinates in the x-axis

-OR-

(∂E/∂w) times step is a small shift on f(w) not w.so why does the difference between W(n+1) and Wn be equal to f(W)

I found a similar question some-confusion-of-gradient-descent, but the accepted answer doesnot have a proof.

$\endgroup$
  • $\begingroup$ Welcome to AI.SE! I'm sorry, this site is for social/conceptual/academic aspects of artificial intelligence as opposed to mathematical/implementation issues. For AI-related math, Cross Validated might be able to help; for pure math, Mathematics is the place to go. For more info on our site, see the tour. $\endgroup$ – Ben N Jan 22 '18 at 22:25
2
$\begingroup$

Don’t think about it as the enter image description here being proportional to something. Think about it this way:

I’m now at enter image description here. Where do I want to be at Time step so that the error decreases? For that, I need to know how the error changes when I make small steps to the left or right of enter image description here If enter image description here increases as I increase enter image description here (that is, if enter image description here, then obviously, I would want to move a little bit to the left. In other words, enter image description here or enter image description here.

On the other hand, if the derivative were negative, you know that you should move right to reduce the error a little bit, enter image description here. So, basically your step should have the opposite sign of the derivative.

enter image description here

enter image description here, the learning rate, is just the constant of proportionality. Caution: think about small values for this rate, not big numbers. Taking a huge step can cause you to overshoot the minimum point.

$\endgroup$
  • $\begingroup$ Oops, no formatting, unlike Math SE. I hope you are able to understand, though. $\endgroup$ – Mathemagical Jan 21 '18 at 11:42
  • $\begingroup$ i have updated the question while you were answering, can you just it once more, TIA $\endgroup$ – souparno majumder Jan 21 '18 at 11:43
  • $\begingroup$ that brings me to another question, if changing the w changes the error, then why cant i write wn - wn+1 be proportional to E? insted why do i need a derivative of E? $\endgroup$ – souparno majumder Jan 21 '18 at 12:10
  • $\begingroup$ @souparnomajumder proportional to E will not work. The bigger the error, you want to move more to the right? Imagine yourself at w=2 on your graph. Moving to the right will give you even more error! $\endgroup$ – Mathemagical Jan 21 '18 at 13:09
  • $\begingroup$ can you check if the following derivation is correct ? mathcha.io/editor/8NmjHWQUxztpofYO $\endgroup$ – souparno majumder Jan 21 '18 at 14:43
0
$\begingroup$

$$let,\ x_{(n)} \ be\ a\ point\ on\ x-axis\ where\ f'( x) \ =\ 0\ ,\ and\ x_{(n\ +\ h)} \ is\ any\ other\ arbitary\ point \\ \therefore \ \ \frac{f'( x_{(n\ +\ h)})}{|\ f'( x_{(n\ +\ h)}) \ |} =\begin{cases} 1 & \mathrm{if,} \ h\ \ >0\\ 0 & \mathrm{if,} \ h\ =\ 0\\ -1 & \mathrm{if,} \ h\ < \ 0 \end{cases}\\similarly,\ \ \frac{x_{(n)} \ -\ x_{(n\ +\ h)} \ }{|\ x_{(n)} \ -\ x_{(n\ +\ h)} \ |} \ =\ \begin{cases} 1 & \mathrm{if,} \ h\ \ < 0\\ 0 & \mathrm{if,} \ h\ =\ 0\\ -1 & \mathrm{if,} \ h\ >\ 0 \end{cases}\\or,\ \frac{x_{(n)} \ -\ x_{(n\ +\ h)} \ }{|\ x_{(n)} \ -\ x_{(n\ +\ h)} \ |} \ =\ -\ \ \ \frac{f'( x_{(n\ +\ h)})}{|\ f'( x_{(n\ +\ h)}) \ |}\\ \therefore \ x_{(n)} \ =x_{(n\ +\ h)} \ -\ \eta \times f'( x_{(n\ +\ h)}) \ \ \ \ \ \ \left[ where\ \eta \ =\frac{|\ x_{(n)} \ -\ x_{(n\ +\ h)} \ |}{|f'( x_{(n\ +\ h)}) \ |} \ \right]$$

$\endgroup$
  • $\begingroup$ I do not think you have proven anything here. In the equation second from bottom m and dy/dx cancel each out, right? What remains is x(n+h)-x(n)=h, which is kind of trivial, i.e. follows from notations you’ve chosen in (I) and (II). $\endgroup$ – aivanov Mar 21 '18 at 21:38

Not the answer you're looking for? Browse other questions tagged or ask your own question.