I am trying to understand the mathematics behind the forward and backward propagation of neural nets. To make myself more comfortable, I am testing myself with an arbitrarily chosen neural network. However, I am stuck at some point.

Consider a simple fully connected neural network with two hidden layers. For simplicity, choose linear activation function (${f(x) = x}$) at all layer. Now consider that this neural network takes two $n$-dimensional inputs $X^{1}$ and $X^{2}$. However, the first hidden layer only takes $X^1$ as the input and produces the output of $H^1$. The second hidden layer takes $H^{1} $and $X^2$ as the input and produces the output $H^{2}$. The output layer takes $H^{2}$ as the input and produces the output $\hat{Y}$. For simplicity, assume, we do not have any bias.

So, we can write that, $H^1 = W^{x1}X^{1}$

$H^2 = W^{h}H1 + W^{x2}X^{2} = W^{h}W^{x1}X^{1} + W^{x2}X^{2}$ [substituting the value of $H^1$]

$\hat{Y} = W^{y}H^2$

Here, $W^{x1}$, $W^{x2}$, $W^{h}$ and $W^{y}$ are the weight matrix. Now, to make it more interesting, consider a sharing weight matrix $W^{x} = W^{x1} = W^{x2}$, which leads, $H^1 = W^{x}X^{1}$ and $H^2 = W^{h}W^{x}X^{1} + W^{x}X^{2}$

I do not have any problem to do forward propagation by my hand; however, the problem arises when I tried to make backward propagation and update the $W^{x}$.

$\frac{\partial loss}{\partial W^{x}} = \frac{\partial loss}{\partial H^{2}} . \frac{\partial H^{2}}{\partial W^{x}}$

Substituting, $\frac{\partial loss}{\partial H^{2}} = \frac{\partial Y}{\partial H^{2}}. \frac{\partial loss}{\partial Y}$ and $H^2 = W^{h}W^{x}X^{1} + W^{x}X^{2}$

$\frac{\partial loss}{\partial W^{x}}= \frac{\partial Y}{\partial H^{2}}. \frac{\partial loss}{\partial Y} . \frac{\partial}{\partial W^{x}} (W^{h}W^{x}X^{1} + W^{x}X^{2})$

Here I understand that, $\frac{\partial Y}{\partial H^{2}} = (W^y)^T$ and $\frac{\partial}{\partial W^{x}} W^{x}X^{2} = (X^{2})^T$ and we can also calculate $\frac{\partial Y}{\partial H^{2}}$, if we know the loss function. But how do we calculate $\frac{\partial}{\partial W^{x}} W^{h}W^{x}X^{1}$?


3 Answers 3


If we write $ H^2 = W^{h}H1 + W^{x}X^{2} $ then it will be better to understand the backward propagation step.


$\frac{\partial}{\partial W^{x}} W^{h}W^{x}X^{1}$ can be written as: $\frac{\partial H^2}{\partial H^1}\frac{\partial H^1}{\partial W^{x}} $

$\frac{\partial H^2}{\partial H^1} = (W^h)^T$ and $\frac{\partial H^1}{\partial W^{x}} = (X^{1})^T $


$\frac{\partial}{\partial W^{x}} W^{h}W^{x}X^{1} = (W^h)^T(X^{1})^T $

I hope it has solved your problem.

  • $\begingroup$ So, what if we have 3 nodes in the hidden layers (i.e., $H^1$ and $H^2$ should be 3x1 vector) and 2-dimensional inputs ($X^1$ and $X^2$) then $W^h$ is a 3x3 weight matrix and $X^1$ is a 2x1 vector, How can we multiply $(W^h)^T$ and $(X^1)^T$? $\endgroup$ May 1, 2020 at 19:29
  • $\begingroup$ Can you please tell me, in this expression $\frac{\partial}{\partial W^{x}} W^{x}X^{2} = (X^{2})^T$, why have you taken transpose of $(X^2)^T$ as derivative. Why is it not only $(X^2)$ $\endgroup$
    – VIJAY
    May 2, 2020 at 3:57

I think your notations are unclear, but I can give an answer based on what you probably meant. For example, $\frac{\partial{L}}{\partial{W^x}}$ should be replaced by $(\nabla_{W^x_{j:}}L)_{j=1, ...,n}$ (assuming everything stays in $\mathbb{R}^n$). Also your expression for $\frac{\partial{L}}{\partial{W^x}}$ is wrong, even accounting for the notation.

Since $W^x_{j:}$ affects the loss through $H_{1,j}$ and $H_{2,j}$, it would be better to treat the math in this way: $$\nabla_{W^x_{j:}}L=\frac{\partial{L}}{\partial{H_{1,j}}}\nabla_{W^x_{j:}}H_{1,j}+\frac{\partial{L}}{\partial{H_{2,j}}}\nabla_{W^x_{j:}}H_{2,j}$$ Now, $H_{1, j}$ affects the loss though $H_{2,k}\ \forall\ k=1,...,n.$ So, $$\frac{\partial{L}}{\partial{H_{1,j}}}=\sum_{k=1}^{n}\frac{\partial{L}} {\partial{H_{2,k}}}W^x_{kj}$$ And, $$\frac{\partial{L}}{\partial{H_{2,j}}}=\sum_{k=1}^{n}\frac{\partial{L}} {\partial{Y_{k}}}W^y_{kj}$$ Similarly, $\nabla_YL$ can be computed.


The product rule of partial derivative:
$\frac{\partial}{\partial x} f g = g \frac{\partial}{\partial x} f + f \frac{\partial}{\partial x} g$

According to this: $\frac{\partial}{\partial W^{x}} W^{h}W^{x}X^{1} = W^{h}X^{1}$, because derivative of other term with respect to $W^{x}$ is zero. (I am not considering the transpose notation as it depends on how you organize your data.)

However, Your assumption of giving $H^{1}$ and $X^{2}$ as input to second hidden layer is not valid(they are called hidden layer for that reason). The output of first hidden layer ($H^{1}$) will be fed to the input of second hidden layer. Your output of second hidden layer would be $H^{2} = W^{h} * H^{1}$.

You have to fed your input $X^{1} X^{2}$ to your network at once by means of looping or vectorization.

  • $\begingroup$ Firstly, what if we have 3 nodes in the hidden layers (i.e., $H^1$ and $H^2$ should be 3x1 vector) and 2-dimensional inputs ($X^1$ and $X^2$) then $W^h$ is a 3x3 weight matrix and $X^1$ is a 2x1 vector, How can we multiply $(W^h)^T$ and $(X^1)^T$? Secondly, think about an Elman RNN network. you can think $X^1$ as input at time t=0 and $X^2$ as input at time t=1. Then the $\hat(Y)$ is the output from the time step t=1 $\endgroup$ May 1, 2020 at 19:47
  • $\begingroup$ My ans is saying what will be derivative of your asked partial derivative equation. The dimension mismatch is happening because the equation you are getting from forward pass is probably wrong. $H^{2} = W^{h} W^{x} X^{1} + W^{x} X^{2}$ is not right i guess. I don't know much about Elman RNN. My ans is based on simple fully connected N.N as you have mentioned in the question. ** adding a diagram would make people better understand your query. Thanks. $\endgroup$
    – tahiat
    May 1, 2020 at 20:12

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .