I am trying to understand the mathematics behind the forward and backward propagation of neural nets. To make myself more comfortable, I am testing myself with an arbitrarily chosen neural network. However, I am stuck at some point.

Consider a simple fully connected neural network with two hidden layers. For simplicity, choose linear activation function (${f(x) = x}$) at all layer. Now consider that this neural network takes two $n$-dimensional inputs $X^{1}$ and $X^{2}$. However, the first hidden layer only takes $X^1$ as the input and produces the output of $H^1$. The second hidden layer takes $H^{1} $and $X^2$ as the input and produces the output $H^{2}$. The output layer takes $H^{2}$ as the input and produces the output $\hat{Y}$. For simplicity, assume, we do not have any bias.

So, we can write that, $H^1 = W^{x1}X^{1}$

$H^2 = W^{h}H1 + W^{x2}X^{2} = W^{h}W^{x1}X^{1} + W^{x2}X^{2}$ [substituting the value of $H^1$]

$\hat{Y} = W^{y}H^2$

Here, $W^{x1}$, $W^{x2}$, $W^{h}$ and $W^{y}$ are the weight matrix. Now, to make it more interesting, consider a sharing weight matrix $W^{x} = W^{x1} = W^{x2}$, which leads, $H^1 = W^{x}X^{1}$ and $H^2 = W^{h}W^{x}X^{1} + W^{x}X^{2}$

I do not have any problem to do forward propagation by my hand; however, the problem arises when I tried to make backward propagation and update the $W^{x}$.

$\frac{\partial loss}{\partial W^{x}} = \frac{\partial loss}{\partial H^{2}} . \frac{\partial H^{2}}{\partial W^{x}}$

Substituting, $\frac{\partial loss}{\partial H^{2}} = \frac{\partial Y}{\partial H^{2}}. \frac{\partial loss}{\partial Y}$ and $H^2 = W^{h}W^{x}X^{1} + W^{x}X^{2}$

$\frac{\partial loss}{\partial W^{x}}= \frac{\partial Y}{\partial H^{2}}. \frac{\partial loss}{\partial Y} . \frac{\partial}{\partial W^{x}} (W^{h}W^{x}X^{1} + W^{x}X^{2})$

Here I understand that, $\frac{\partial Y}{\partial H^{2}} = (W^y)^T$ and $\frac{\partial}{\partial W^{x}} W^{x}X^{2} = (X^{2})^T$ and we can also calculate $\frac{\partial Y}{\partial H^{2}}$, if we know the loss function. But how do we calculate $\frac{\partial}{\partial W^{x}} W^{h}W^{x}X^{1}$?


If we write $ H^2 = W^{h}H1 + W^{x}X^{2} $ then it will be better to understand the backward propagation step.


$\frac{\partial}{\partial W^{x}} W^{h}W^{x}X^{1}$ can be written as: $\frac{\partial H^2}{\partial H^1}\frac{\partial H^1}{\partial W^{x}} $

$\frac{\partial H^2}{\partial H^1} = (W^h)^T$ and $\frac{\partial H^1}{\partial W^{x}} = (X^{1})^T $


$\frac{\partial}{\partial W^{x}} W^{h}W^{x}X^{1} = (W^h)^T(X^{1})^T $

I hope it has solved your problem.

  • $\begingroup$ So, what if we have 3 nodes in the hidden layers (i.e., $H^1$ and $H^2$ should be 3x1 vector) and 2-dimensional inputs ($X^1$ and $X^2$) then $W^h$ is a 3x3 weight matrix and $X^1$ is a 2x1 vector, How can we multiply $(W^h)^T$ and $(X^1)^T$? $\endgroup$ May 1 '20 at 19:29
  • $\begingroup$ Can you please tell me, in this expression $\frac{\partial}{\partial W^{x}} W^{x}X^{2} = (X^{2})^T$, why have you taken transpose of $(X^2)^T$ as derivative. Why is it not only $(X^2)$ $\endgroup$
    – VIJAY
    May 2 '20 at 3:57

The product rule of partial derivative:
$\frac{\partial}{\partial x} f g = g \frac{\partial}{\partial x} f + f \frac{\partial}{\partial x} g$

According to this: $\frac{\partial}{\partial W^{x}} W^{h}W^{x}X^{1} = W^{h}X^{1}$, because derivative of other term with respect to $W^{x}$ is zero. (I am not considering the transpose notation as it depends on how you organize your data.)

However, Your assumption of giving $H^{1}$ and $X^{2}$ as input to second hidden layer is not valid(they are called hidden layer for that reason). The output of first hidden layer ($H^{1}$) will be fed to the input of second hidden layer. Your output of second hidden layer would be $H^{2} = W^{h} * H^{1}$.

You have to fed your input $X^{1} X^{2}$ to your network at once by means of looping or vectorization.

  • $\begingroup$ Firstly, what if we have 3 nodes in the hidden layers (i.e., $H^1$ and $H^2$ should be 3x1 vector) and 2-dimensional inputs ($X^1$ and $X^2$) then $W^h$ is a 3x3 weight matrix and $X^1$ is a 2x1 vector, How can we multiply $(W^h)^T$ and $(X^1)^T$? Secondly, think about an Elman RNN network. you can think $X^1$ as input at time t=0 and $X^2$ as input at time t=1. Then the $\hat(Y)$ is the output from the time step t=1 $\endgroup$ May 1 '20 at 19:47
  • $\begingroup$ My ans is saying what will be derivative of your asked partial derivative equation. The dimension mismatch is happening because the equation you are getting from forward pass is probably wrong. $H^{2} = W^{h} W^{x} X^{1} + W^{x} X^{2}$ is not right i guess. I don't know much about Elman RNN. My ans is based on simple fully connected N.N as you have mentioned in the question. ** adding a diagram would make people better understand your query. Thanks. $\endgroup$
    – tahiat
    May 1 '20 at 20:12

I think your notations are unclear, but I can give an answer based on what you probably meant. For example, $\frac{\partial{L}}{\partial{W^x}}$ should be replaced by $(\nabla_{W^x_{j:}}L)_{j=1, ...,n}$ (assuming everything stays in $\mathbb{R}^n$). Also your expression for $\frac{\partial{L}}{\partial{W^x}}$ is wrong, even accounting for the notation.

Since $W^x_{j:}$ affects the loss through $H_{1,j}$ and $H_{2,j}$, it would be better to treat the math in this way: $$\nabla_{W^x_{j:}}L=\frac{\partial{L}}{\partial{H_{1,j}}}\nabla_{W^x_{j:}}H_{1,j}+\frac{\partial{L}}{\partial{H_{2,j}}}\nabla_{W^x_{j:}}H_{2,j}$$ Now, $H_{1, j}$ affects the loss though $H_{2,k}\ \forall\ k=1,...,n.$ So, $$\frac{\partial{L}}{\partial{H_{1,j}}}=\sum_{k=1}^{n}\frac{\partial{L}} {\partial{H_{2,k}}}W^x_{kj}$$ And, $$\frac{\partial{L}}{\partial{H_{2,j}}}=\sum_{k=1}^{n}\frac{\partial{L}} {\partial{Y_{k}}}W^y_{kj}$$ Similarly, $\nabla_YL$ can be computed.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.