Why do we use gradient descent to minimize the loss function?

Question

The purpose of training neural networks is to minimize a loss function, in this process we usually use gradient descent method.

But in Calculus, if we want to find the global minimum of a multivariable function, we usually first calculate the partial derivatives of this function with respect to its variables, and then set these partial derivatives to zeros, and then find the solutions of these equations. Usually we get a bunch of points. We can use the second derivative method(involving Hesse Matrix) to determine whether these points give local minimum values, or we can directly evaluate the values of the function at these points and compare them to find the minimum.

So, I'm curious why don't we use this classical method in Calculus to find the global minimum value of the loss function and instead using gradient descent? Is it hard to for computer to find zeros of multivariable equations?

Answer 1

and then set these partial derivatives to zeros

but how do you do that? This is analytically possible in very simple cases, such as least squares on linear regression. In that case, it's possible to invert the function and explicitly build the weight matrix and the biases.

But in maths, the vast majority of equations is not analytically solvable, and needs iterative approximation methods that lead you to a numerical solution. This is what gradient descent does.

To convince yourself of this, I suggest you try: write a simple function representing a neural network with 2 layers, then see if you can do as you say. This will give you a better understanding of why it doesn't work.