# Isssue in understanding the derivation regarding mean squared error

The following derivation is taken from Chapter 5: Machine Learning Basics from the book titled Deep Learning (by Aaron Courville et al.)

I am facing difficulty in understanding the zero derivative related to minimizing gradient of mean squared error $$\nabla_w \text{MSE}_{train} = 0$$

$$\nabla_w \text{MSE}_{train} = 0$$

$$\implies \nabla_w(Xw - y)^T(Xw-y) = 0$$

$$\implies \nabla_w(w^T X^T- y^T)(Xw-y) = 0$$

$$\implies \nabla_w(w^T X^T- y^T)(Xw-y) = 0$$

$$\implies \nabla_w(w^T X^TXw - w^T X^Ty -y^TXw+y^Ty) = 0$$

$$\implies \nabla_w(w^T X^TXw - 2 w^T X^Ty +y^Ty) = 0$$

$$\implies 2 X^TXw - 2 X^Ty = 0$$

$$\implies w = 2 (X^TX)^{-1} X^Ty = 0$$

I have had difficulty in understanding the flow of the following two lines

$$\implies \nabla_w(w^T X^TXw - w^T X^Ty -y^TXw+y^Ty) = 0$$

$$\implies \nabla_w(w^T X^TXw - 2 w^T X^Ty +y^Ty) = 0$$

The first doubt is about in the first two lines, it is possible only if (which I feel is untrue)

$$w^T X^Ty = y^TXw$$

Note: $$X,y$$ here refers to input and outputs of train data

1. If you have a scalar $$x \in \mathbb{R}$$, then you can take its transpose without changing the value, its transpose is exactly the same number.
2. When you have two vectors $$w \in \mathbb{R}^n$$ and $$y \in \mathbb{R}^n$$ and a matrix $$X \in \mathbb{R}^{n \times n}$$, then $$w^TX^Ty \in \mathbb{R}$$, i.e. it is a scalar.
3. From Linear Algebra, when you have a product of matrices ($$A$$, $$B$$, etc. matrices) you can take the transpose as follows: $$(AB)^T = B^TA^T$$. You can extend that two three matrices: $$(ABC)^T = ((AB)C)^T = C^T(AB)^T = C^TB^TA^T$$. You can look at vectors and scalars as special kinds of matrices.
4. When you have $$w^TX^Ty$$, you can take its transpose without changing the value because it is a scalar, so $$w^TX^Ty = (w^TX^Ty)^T = y^TXw$$.