# Understanding the math behind using maximum likelihood for linear regression

I understand both terms, linear regression and maximum likelihood, but, when it comes to the math, I am totally lost. So I am reading this article The Principle of Maximum Likelihood (by Suriyadeepan Ramamoorthy). It is really well written, but, as mentioned in the previous sentence, I don't get the math.

The joint probability distribution of $$y,\theta, \sigma$$ is given by (assuming $$y$$ is normally distributed):

This equivalent to maximizing the log likelihood:

The maxima can be then equating through the derivative of l(θ) to zero:

I get everything until this point, but don't understand how this function is equivalent to the previous one :

• This kind of question goes well on Maths Stack Exchange. – samerivertwice Nov 27 '18 at 12:02
• @RobertFrost I think this type of questions can be asked here, given the topic (maximum likelihood and linear regression). AI is still a mathematical-based field, so this type of questions are normal in the AI field too. Honestly, I would like to see more questions of this kind here. – nbro Nov 27 '18 at 12:16
• @nbro me too, I was just saying in case you wanted more. – samerivertwice Nov 27 '18 at 15:24

Note first that the first $$=$$ (equals) in $$\frac{dl(\theta)}{d\theta} = 0 = −\frac{1}{2\sigma^2}(0−2X^TY + X^TX \theta)$$ should be interpreted as a "is set to", that is, we set $$\frac{dl(\theta)}{d\theta} = 0$$. Given that (apparently) $$\frac{dl(\theta)}{d\theta} = −\frac{1}{2\sigma^2}(0−2X^TY + X^TX \theta)$$, $$\frac{dl(\theta)}{d\theta} = 0$$ is equivalent to $$0 = −\frac{1}{2\sigma^2}(0−2X^TY + X^TX \theta)$$.

Now, let's apply some basic linear algebra:

\begin{align} 0 &= −\frac{1}{2\sigma^2}(0−2X^TY + X^TX \theta) \iff \\ 0 &= −(0−2X^TY + X^TX \theta) \iff \\ 0 &= −0 + 2X^TY - X^TX \theta) \iff \\ 0 &= 2X^TY - X^TX \theta \iff \\ X^TX \theta &= 2X^TY \iff \\ (X^TX)^{-1}(X^TX) \theta &= (X^TX)^{-1}2X^TY \iff \\ \theta &= (X^TX)^{-1}2X^TY \end{align}

Now, you can ignore the $$2$$, because it is just a constant, and, when optimizing, this does not influence the result.

Note that using $$\hat{\theta}$$ instead of $$\theta$$ is just to indicate that what we will get is an "estimate" of the real $$\theta$$, because of round off errors during the computations, etc.

• I think it would be better to clarify under what circumstances and why, constants can be ignored. It looks at face value like doing so will halve the value of theta. – samerivertwice Nov 27 '18 at 3:26
• @RobertFrost I think there's a wrong derivation in the original article and that 2 would not even occur in the calculations above anymore, because it would cancel out with the 2 in the denominator. Essentially, $(Y - X\theta)^T(Y - X\theta)$ would have a term $X^TX\theta^2$ and the partial derivative of $l$ with respect to $\theta$ would produce the term $2X^TX\theta$ (but the original author of the article left out the 2 in front). In that case, after the partial derivation, all addends would have a $2$ in front, which would cancel out with the 2 at the denominator. – nbro Nov 27 '18 at 10:32
• Maybe that's what confused the OP – samerivertwice Nov 27 '18 at 12:01
• @RobertFrost I think what confused the OP was that the "$= 0$" is actually "setting to $0$", but only the OP can tell us. Maybe more than one thing confused him/her. – nbro Nov 27 '18 at 12:14
• Well , in fact, I was confused about the disappearance of 2 in the last equation. thank you very much for the detailed answer! – xava Nov 27 '18 at 13:25