0
$\begingroup$

Let's suppose we have calculated the gradient and it came out to be $f(WX)(1-f(W X))X$, where $f()$ is the sigmoid function, $W$ of order $2\times2$ is the weight matrix, and $X$ is an input vector of order $2\times 1$. For ease let $f(WX)(1-f(W X))=\Bigg[ \begin{array}{c} 0.3 \\ 0.8 \\ \end{array}\Bigg]$ and $X=\Bigg[ \begin{array}{c} 1 \\ 0 \\ \end{array}\Bigg]$. When we multiply these vectors we will multiply them as $f(WX)(1-f(W X))\times X^T$ i.e $\Bigg[ \begin{array}{c} 0.3 \\ 0.8 \\ \end{array}\Bigg]\times[1 \quad0]$. I do this because I know that we need this gradient to update a $2\times 2$ weight matrix, hence, the gradient should have size $2\times 2$. But, I don't know the law/rule behind this, if I was just given the values and had no knowledge that we need the solution to update the weight matrix, then, I might have done something like $[0.3 \quad 0.8]\times\Bigg[ \begin{array}{c} 1 \\ 0 \\ \end{array}\Bigg]$ which will return a scalar. For a long chain of such operations (multiple derivatives in applying chain rule, resulting in many vectors), how do we know if the multiplication of two vectors should return a vector or matrix (dot or cross product)?

$\endgroup$

1 Answer 1

0
$\begingroup$

It helps to think of each output dimension separately.

You have $X$ which is an $(2 \times 1)$ vector, and $W_1$ is a $(1 \times 2 )$ vector. Their product is a scalar, of which we then take the sigmoid to get our output $Y_1$.

The gradient of this w.r.t. $W_1$ will be $f(WX) (1 - f(WX)) X^T$, which has the appropriate dimensions.

Then, you just stack these for all your outputs, giving you the shape you got.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .