Recently, I followed the open course CS229, http://cs229.stanford.edu/notes/cs229-notes1.pdf
This lecturer introduces an alternative approach to gradient descent that is called "Normal Equation" and the equation is as follows:

$$\theta=\left(X^{T} X\right)^{-1} X^{T} \vec{y}$$

The normal equation can directly compute the $\theta$.

If the normal equation works, why do we need gradient descent? What is the trade-off between these two methods?


That normal equation is sometimes called the closed-form solution.

The short answer to your question is that the closed-form solution may be impractical or unavailable in certain cases or the iterative numerical method (such as gradient descent) may be more efficient (in terms of resources).

This answer gives you more details and an example.

| improve this answer | |
  • 1
    $\begingroup$ Thank you nbro, you really helped me out! $\endgroup$ – Cheleeger Ken Jul 8 at 14:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.