If the normal equation works, why do we need gradient descent?

Question

Recently, I followed the open course CS229, http://cs229.stanford.edu/notes/cs229-notes1.pdf
This lecturer introduces an alternative approach to gradient descent that is called "Normal Equation" and the equation is as follows:

$$\theta=\left(X^{T} X\right)^{-1} X^{T} \vec{y}$$

The normal equation can directly compute the $\theta$.

If the normal equation works, why do we need gradient descent? What is the trade-off between these two methods?

Answer 1

That normal equation is sometimes called the closed-form solution.

The short answer to your question is that the closed-form solution may be impractical or unavailable in certain cases or the iterative numerical method (such as gradient descent) may be more efficient (in terms of resources).

This answer gives you more details and an example.