2
$\begingroup$

I understand both terms, linear regression and maximum likelihood, but, when it comes to the math, I am totally lost. So I am reading this article The Principle of Maximum Likelihood (by Suriyadeepan Ramamoorthy). It is really well written, but, as mentioned in the previous sentence, I don't get the math.

The joint probability distribution of $y,\theta, \sigma$ is given by (assuming $y$ is normally distributed):

joint probability distribution of y,θ,σ

This equivalent to maximizing the log likelihood: enter image description here

The maxima can be then equating through the derivative of l(θ) to zero: enter image description here

I get everything until this point, but don't understand how this function is equivalent to the previous one : enter image description here

$\endgroup$
3
  • $\begingroup$ This kind of question goes well on Maths Stack Exchange. $\endgroup$ Nov 27, 2018 at 12:02
  • 1
    $\begingroup$ @RobertFrost I think this type of questions can be asked here, given the topic (maximum likelihood and linear regression). AI is still a mathematical-based field, so this type of questions are normal in the AI field too. Honestly, I would like to see more questions of this kind here. $\endgroup$
    – nbro
    Nov 27, 2018 at 12:16
  • $\begingroup$ @nbro me too, I was just saying in case you wanted more. $\endgroup$ Nov 27, 2018 at 15:24

1 Answer 1

2
$\begingroup$

Note first that the first $=$ (equals) in $\frac{dl(\theta)}{d\theta} = 0 = −\frac{1}{2\sigma^2}(0−2X^TY + X^TX \theta)$ should be interpreted as a "is set to", that is, we set $\frac{dl(\theta)}{d\theta} = 0$. Given that (apparently) $\frac{dl(\theta)}{d\theta} = −\frac{1}{2\sigma^2}(0−2X^TY + X^TX \theta)$, $\frac{dl(\theta)}{d\theta} = 0$ is equivalent to $0 = −\frac{1}{2\sigma^2}(0−2X^TY + X^TX \theta)$.

Now, let's apply some basic linear algebra:

\begin{align} 0 &= −\frac{1}{2\sigma^2}(0−2X^TY + X^TX \theta) \iff \\ 0 &= −(0−2X^TY + X^TX \theta) \iff \\ 0 &= −0 + 2X^TY - X^TX \theta) \iff \\ 0 &= 2X^TY - X^TX \theta \iff \\ X^TX \theta &= 2X^TY \iff \\ (X^TX)^{-1}(X^TX) \theta &= (X^TX)^{-1}2X^TY \iff \\ \theta &= (X^TX)^{-1}2X^TY \end{align}

Now, you can ignore the $2$, because it is just a constant, and, when optimizing, this does not influence the result.

Note that using $\hat{\theta}$ instead of $\theta$ is just to indicate that what we will get is an "estimate" of the real $\theta$, because of round off errors during the computations, etc.

$\endgroup$
5
  • $\begingroup$ I think it would be better to clarify under what circumstances and why, constants can be ignored. It looks at face value like doing so will halve the value of theta. $\endgroup$ Nov 27, 2018 at 3:26
  • 1
    $\begingroup$ @RobertFrost I think there's a wrong derivation in the original article and that 2 would not even occur in the calculations above anymore, because it would cancel out with the 2 in the denominator. Essentially, $(Y - X\theta)^T(Y - X\theta)$ would have a term $X^TX\theta^2$ and the partial derivative of $l$ with respect to $\theta$ would produce the term $2X^TX\theta$ (but the original author of the article left out the 2 in front). In that case, after the partial derivation, all addends would have a $2$ in front, which would cancel out with the 2 at the denominator. $\endgroup$
    – nbro
    Nov 27, 2018 at 10:32
  • $\begingroup$ Maybe that's what confused the OP $\endgroup$ Nov 27, 2018 at 12:01
  • $\begingroup$ @RobertFrost I think what confused the OP was that the "$= 0$" is actually "setting to $0$", but only the OP can tell us. Maybe more than one thing confused him/her. $\endgroup$
    – nbro
    Nov 27, 2018 at 12:14
  • 2
    $\begingroup$ Well , in fact, I was confused about the disappearance of 2 in the last equation. thank you very much for the detailed answer! $\endgroup$
    – xava
    Nov 27, 2018 at 13:25

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .