0
$\begingroup$

The following derivation is taken from Chapter 5: Machine Learning Basics from the book titled Deep Learning (by Aaron Courville et al.)

I am facing difficulty in understanding the zero derivative related to minimizing gradient of mean squared error $\nabla_w \text{MSE}_{train} = 0$

$\nabla_w \text{MSE}_{train} = 0$

$\implies \nabla_w(Xw - y)^T(Xw-y) = 0$

$\implies \nabla_w(w^T X^T- y^T)(Xw-y) = 0$

$\implies \nabla_w(w^T X^T- y^T)(Xw-y) = 0$

$\implies \nabla_w(w^T X^TXw - w^T X^Ty -y^TXw+y^Ty) = 0$

$\implies \nabla_w(w^T X^TXw - 2 w^T X^Ty +y^Ty) = 0$

$\implies 2 X^TXw - 2 X^Ty = 0$

$\implies w = 2 (X^TX)^{-1} X^Ty = 0$

I have had difficulty in understanding the flow of the following two lines

$\implies \nabla_w(w^T X^TXw - w^T X^Ty -y^TXw+y^Ty) = 0$

$\implies \nabla_w(w^T X^TXw - 2 w^T X^Ty +y^Ty) = 0$

The first doubt is about in the first two lines, it is possible only if (which I feel is untrue)

$w^T X^Ty = y^TXw$

Note: $X,y$ here refers to input and outputs of train data

$\endgroup$
0
1
$\begingroup$

The topic you're looking for is called Linear Algebra, and the equation you highlighted at the end is true. This is the reason:

  1. If you have a scalar $x \in \mathbb{R}$, then you can take its transpose without changing the value, its transpose is exactly the same number.
  2. When you have two vectors $w \in \mathbb{R}^n$ and $y \in \mathbb{R}^n$ and a matrix $X \in \mathbb{R}^{n \times n}$, then $w^TX^Ty \in \mathbb{R}$, i.e. it is a scalar.
  3. From Linear Algebra, when you have a product of matrices ($A$, $B$, etc. matrices) you can take the transpose as follows: $(AB)^T = B^TA^T$. You can extend that two three matrices: $(ABC)^T = ((AB)C)^T = C^T(AB)^T = C^TB^TA^T$. You can look at vectors and scalars as special kinds of matrices.
  4. When you have $w^TX^Ty$, you can take its transpose without changing the value because it is a scalar, so $w^TX^Ty = (w^TX^Ty)^T = y^TXw$.

Most books and free online resources on Linear Algebra are enough to understand these concepts. Everyone has their favourite, and it really doesn't matter much which one you pick but I like this book: https://link.springer.com/book/10.1007/978-3-319-24346-7.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.