How parameter adjustment works in Gradient Descent?

Question

I am trying to comprehend how the Gradient Descent works.

I understand we have a cost function which is defined in terms of the following parameters,

$J(𝑤_{1},𝑤_{2},.... , w_{n}, b)$

the derivative would tell us which direction to adjust the parameters.

i.e. $\dfrac{dJ(𝑤_{1},𝑤_{2},.... , w_{n}, b)}{d(𝑤_{1}}$ is the rate of change of the cost w.r.t $𝑤$

The lecture kept on saying this very valuable as we are asking the question how should I change $𝑤$ to improve the cost?

But then the Lecturer presented $w_{1}$, $w_{2}$, ... as scaler value, How can we differentiate a scalar value.

I am fundamentally missing what is happening.

Can anyone please guide me to any blog post, a book that I should read to understand better?

Answer 1

Imagine we have the curve $f(x) = x^2$, and we want to find the minimum of this function. The derivate of $f$ with respect to $x$ is $2x$. Now, gradient descent works by updating our current estimate of the minimum, say $c_t$, by the following iterative process $$c_{t+1} = c_t - \alpha \times \nabla_xf(x=c_t),$$ where $\alpha$ is some constant to control how much we shift towards the gradient.

Intuitively, this should make sense. Imagine our current estimate of the minimum is $c_t = -1$. The update would then give us $c_{t+1} = -1 - \alpha \times -2 = -1 + 2\alpha > -1$. As you can see, the update has shifted our estimate in the direction of the minimum. If our estimate were $+1$ then you can probably see that the update would again have shifted us in the direction of the minimum again.

Now, what happens in machine learning is we have a loss function $L$ that we typically want to find the minimum of in terms of the parameters of our model. By applying gradient descent on the loss function as we did above with $f(x)$ we iteratively apply the update rule which will eventually lead us to the minimum of our loss function with respect to the weights. The process is exactly the same as above except it is likely to happen in higher dimensions, where the derivative of $f$ becomes a vector of partial derivates. Note that whilst $w_i$'s are scalar values, we are not differentiating these values, rather we are differentiating the loss function with respect to these scalars.

I would recommend you to try this out with a simple linear model of maybe 2 or 3 parameters. After a quick Google I found this article that may be useful.