# Unclear steps in derivation of normal equations in linear regression using linear algebra approach

How are eqs.(3.55) and (3.56) obtained? Especially, it is unclear how triangle inequality implies eq.(3.56) because we have squared norms.

• Please don't use screen shots for text excerpts. It makes the text inaccessible to search on this site. Commented Nov 12, 2023 at 14:07
• Can you provide more context and details of your reference? Seems here is just plain Euclidean norm with standard Minkowski triangle inequality, there shouldn't be and no need of square in the upper right position of all your 2-norms. Commented Nov 13, 2023 at 21:47
• Please see page 61 of: smlbook.org/book/sml-book-draft-latest.pdf Commented Nov 13, 2023 at 22:34

You're right in the linear algebra approach here it's inaccurate to keep all the squared forms of all the 2-norms to arrive at the normal equation of the ordinary least squares (OLS) method for choosing the unknown parameters $$\widehat{\theta}$$ in a linear regression model, and bear in mind that your reference is only a draft version.
Without the squares the same derivation looks ok, on one hand you get $$\|\mathbf{X\theta-y}\|_2=\|\mathbf{(X\theta-y_{\|})-y_{\perp}}\|_2 \geq \|\mathbf{y_{\perp}}\|_2$$, since the same reference reason as $$\mathbf{y_{\perp}}$$ is orthogonal to both $$\mathbf{y_{\|}}$$ and $$\mathbf{X\theta}$$ by their definition. On the other hand, by the same corresponding reference reason as triangle inequality you also get $$\|\mathbf{X\theta-y}\|_2=\|\mathbf{(X\theta-y_{\|})+(-y_{\perp})}\|_2 \leq \|\mathbf{X\theta-y_{\|}}\|_2 + \|\mathbf{-y_{\perp}}\|_2 = \|\mathbf{X\theta-y_{\|}}\|_2 + \|\mathbf{y_{\perp}}\|_2$$
Thus finally you arrive at the same conclusion that if $$\mathbf{X\theta=y_{\|}}$$ the least squared error $$\|\mathbf{X\theta-y}\|_2^2$$ is achieved as $$\|\mathbf{y_{\perp}}\|_2^2$$ and the normal equation (3.13) must hold simultaneously.