# How can the gradient of the weight be calculated in the viewpoint of matrix calculus?

Let $$\sigma(x)$$ be sigmoid function. Consider the case where $$\text{out}=\sigma(\vec{x} \times W + \vec{b})$$, and we want to compute $$\frac{\partial{\text{out}}}{\partial{w} }.$$
Set the dimension as belows:
$$\vec{x}$$: $$(n, n_{\text{in}})$$, $$W$$: $$(n_{\text{in}}, n_{\text{out}})$$, $$\vec{b}$$: $$(1, n_{\text{out}})$$.
Then $$\text{out}$$ has the dimension $$(n, n_{\text{out}})$$. So we need to calculate the matrix by matrix derivative, as I know there is no such way to define that. I know that finally it is calculated as $$\vec{x}^T \times (\text{out}\cdot(1-\text{out}))$$.
But I can't still get the exact procedure of calculation, why it should be $$\vec{x}^T \times (\text{out}\cdot(1-\text{out}))$$, not $$(\text{out}\cdot(1-\text{out})) \times \vec{x}^T$$,I know it by considering dimension, but not by calculation.

My intuition about this problem is that all calculation can be considerd as vector by vector differentiation since $$n$$ is a batch size number, we can calculate matrix differentiation by considering each column vector.

I'm not sure about my intuition yet, and I need some exact mathematical calculation procedure for the problem,

• What is $n$ in your context? The number of samples/examples in your batch that you pass to the net? When you say "I know that finally it is calculated as...", what exactly are you talking about? Do you mean the derivative that you want to calculate? Where does that formula come from? – nbro Apr 15 at 12:25
• 1. 𝑛 menas the number of samples. 2. I know that ∂out / ∂𝑤 has that result, because of 'derivative of composition function' and I saw it on my textbook(which is based on some internet resources). It came from the 'derivative of sigmoid w.r.t (xW + b)' * 'derivative of xW + b w.r.t w ( = x.T)'. I'm just wondering that how can matrix-by-matrix derivative be defined, and ∂out / ∂𝑤 have those result. Simply, I can understand it when 𝑛 = 1, but not in other case. Sorry for my english nonfluency. – 최선웅 Apr 19 at 0:51
• I self-answered to my question, will it be helpful for your question? – 최선웅 Apr 19 at 2:10

I think I understand the process now. Let $$x$$ has the dimension $$(n, p)$$ and W has the dimension $$(p, q)$$. In neural network, $$n$$ denotes the samples in the batch and $$p, q$$ denotes input dimension and output dimension of each layer, respectively. We don't use bias $$b$$ here just for simplicity.

I had trouble in understanding the process of differentiation because of some confusion in vector(matrix) derivative. By changing matrix as set of column vector, now I can solve the problem.

Let $$x$$ = $$\begin{bmatrix} x_{11}, &x_{12}, &\cdots, &x_{1p}\\ x_{21}, &x_{22}, &\cdots, &x_{2p}\\ \vdots, &\vdots, &\vdots, &\vdots\\ x_{n1}, &x_{n2}, &\cdots, &x_{np} \end{bmatrix}$$, which can be written as $$\begin{bmatrix} \vec{x}_{1}, &\vec{x}_{2}, &\cdots, &\vec{x}_{p} \end{bmatrix}$$.

Similarly, $$W$$ = $$\begin{bmatrix} w_{11}, &w_{12}, &\cdots, &w_{1q}\\ w_{21}, &w_{22}, &\cdots, &w_{2q}\\ \vdots, &\vdots, &\vdots, &\vdots\\ w_{p1}, &w_{n2}, &\cdots, &w_{pq} \end{bmatrix}$$ = $$\begin{bmatrix} \vec{w}_{1}\\ \vec{w}_{2}, \\ \vdots, \\ \vec{w}_{p} \end{bmatrix}$$.

Then $$x\times W$$ = $$\vec{x}_1 \vec{w}_1 + \vec{x}_2 \vec{w}_2 + \cdots \vec{x}_p \vec{w}_p$$. So $$\frac{\partial(\vec{x}_1 \vec{w}_1 + \vec{x}_2 \vec{w}_2 + \cdots \vec{x}_p \vec{w}_p)}{\partial{W}} = \begin{bmatrix} \vec{x}_{1}, &\vec{x}_{2}, &\cdots, &\vec{x}_{p} \end{bmatrix}^T.$$

I still have a little trouble in understanding the last part since actually, $$\vec{x}_1 \vec{w}_1 + \vec{x}_2 \vec{w}_2 + \cdots \vec{x}_p \vec{w}_p$$ is not scalar, it's $$(n, q)$$ matrix, but at least now I understand why it is written as above result.

(Also hope to fully understand the last part.)