Backpropagation of neural nets with shared weight

I am trying to understand the mathematics behind the forward and backward propagation of neural nets. To make myself more comfortable, I am testing myself with an arbitrarily chosen neural network. However, I am stuck at some point.

Consider a simple fully connected neural network with two hidden layers. For simplicity, choose linear activation function ($${f(x) = x}$$) at all layer. Now consider that this neural network takes two $$n$$-dimensional inputs $$X^{1}$$ and $$X^{2}$$. However, the first hidden layer only takes $$X^1$$ as the input and produces the output of $$H^1$$. The second hidden layer takes $$H^{1}$$and $$X^2$$ as the input and produces the output $$H^{2}$$. The output layer takes $$H^{2}$$ as the input and produces the output $$\hat{Y}$$. For simplicity, assume, we do not have any bias.

So, we can write that, $$H^1 = W^{x1}X^{1}$$

$$H^2 = W^{h}H1 + W^{x2}X^{2} = W^{h}W^{x1}X^{1} + W^{x2}X^{2}$$ [substituting the value of $$H^1$$]

$$\hat{Y} = W^{y}H^2$$

Here, $$W^{x1}$$, $$W^{x2}$$, $$W^{h}$$ and $$W^{y}$$ are the weight matrix. Now, to make it more interesting, consider a sharing weight matrix $$W^{x} = W^{x1} = W^{x2}$$, which leads, $$H^1 = W^{x}X^{1}$$ and $$H^2 = W^{h}W^{x}X^{1} + W^{x}X^{2}$$

I do not have any problem to do forward propagation by my hand; however, the problem arises when I tried to make backward propagation and update the $$W^{x}$$.

$$\frac{\partial loss}{\partial W^{x}} = \frac{\partial loss}{\partial H^{2}} . \frac{\partial H^{2}}{\partial W^{x}}$$

Substituting, $$\frac{\partial loss}{\partial H^{2}} = \frac{\partial Y}{\partial H^{2}}. \frac{\partial loss}{\partial Y}$$ and $$H^2 = W^{h}W^{x}X^{1} + W^{x}X^{2}$$

$$\frac{\partial loss}{\partial W^{x}}= \frac{\partial Y}{\partial H^{2}}. \frac{\partial loss}{\partial Y} . \frac{\partial}{\partial W^{x}} (W^{h}W^{x}X^{1} + W^{x}X^{2})$$

Here I understand that, $$\frac{\partial Y}{\partial H^{2}} = (W^y)^T$$ and $$\frac{\partial}{\partial W^{x}} W^{x}X^{2} = (X^{2})^T$$ and we can also calculate $$\frac{\partial Y}{\partial H^{2}}$$, if we know the loss function. But how do we calculate $$\frac{\partial}{\partial W^{x}} W^{h}W^{x}X^{1}$$?

If we write $$H^2 = W^{h}H1 + W^{x}X^{2}$$ then it will be better to understand the backward propagation step.

Now,

$$\frac{\partial}{\partial W^{x}} W^{h}W^{x}X^{1}$$ can be written as: $$\frac{\partial H^2}{\partial H^1}\frac{\partial H^1}{\partial W^{x}}$$

$$\frac{\partial H^2}{\partial H^1} = (W^h)^T$$ and $$\frac{\partial H^1}{\partial W^{x}} = (X^{1})^T$$

Therefore,

$$\frac{\partial}{\partial W^{x}} W^{h}W^{x}X^{1} = (W^h)^T(X^{1})^T$$

I hope it has solved your problem.

• So, what if we have 3 nodes in the hidden layers (i.e., $H^1$ and $H^2$ should be 3x1 vector) and 2-dimensional inputs ($X^1$ and $X^2$) then $W^h$ is a 3x3 weight matrix and $X^1$ is a 2x1 vector, How can we multiply $(W^h)^T$ and $(X^1)^T$? May 1 '20 at 19:29
• Can you please tell me, in this expression $\frac{\partial}{\partial W^{x}} W^{x}X^{2} = (X^{2})^T$, why have you taken transpose of $(X^2)^T$ as derivative. Why is it not only $(X^2)$ May 2 '20 at 3:57

The product rule of partial derivative:
$$\frac{\partial}{\partial x} f g = g \frac{\partial}{\partial x} f + f \frac{\partial}{\partial x} g$$

According to this: $$\frac{\partial}{\partial W^{x}} W^{h}W^{x}X^{1} = W^{h}X^{1}$$, because derivative of other term with respect to $$W^{x}$$ is zero. (I am not considering the transpose notation as it depends on how you organize your data.)

However, Your assumption of giving $$H^{1}$$ and $$X^{2}$$ as input to second hidden layer is not valid(they are called hidden layer for that reason). The output of first hidden layer ($$H^{1}$$) will be fed to the input of second hidden layer. Your output of second hidden layer would be $$H^{2} = W^{h} * H^{1}$$.

You have to fed your input $$X^{1} X^{2}$$ to your network at once by means of looping or vectorization.

• Firstly, what if we have 3 nodes in the hidden layers (i.e., $H^1$ and $H^2$ should be 3x1 vector) and 2-dimensional inputs ($X^1$ and $X^2$) then $W^h$ is a 3x3 weight matrix and $X^1$ is a 2x1 vector, How can we multiply $(W^h)^T$ and $(X^1)^T$? Secondly, think about an Elman RNN network. you can think $X^1$ as input at time t=0 and $X^2$ as input at time t=1. Then the $\hat(Y)$ is the output from the time step t=1 May 1 '20 at 19:47
• My ans is saying what will be derivative of your asked partial derivative equation. The dimension mismatch is happening because the equation you are getting from forward pass is probably wrong. $H^{2} = W^{h} W^{x} X^{1} + W^{x} X^{2}$ is not right i guess. I don't know much about Elman RNN. My ans is based on simple fully connected N.N as you have mentioned in the question. ** adding a diagram would make people better understand your query. Thanks. May 1 '20 at 20:12

I think your notations are unclear, but I can give an answer based on what you probably meant. For example, $$\frac{\partial{L}}{\partial{W^x}}$$ should be replaced by $$(\nabla_{W^x_{j:}}L)_{j=1, ...,n}$$ (assuming everything stays in $$\mathbb{R}^n$$). Also your expression for $$\frac{\partial{L}}{\partial{W^x}}$$ is wrong, even accounting for the notation.

Since $$W^x_{j:}$$ affects the loss through $$H_{1,j}$$ and $$H_{2,j}$$, it would be better to treat the math in this way: $$\nabla_{W^x_{j:}}L=\frac{\partial{L}}{\partial{H_{1,j}}}\nabla_{W^x_{j:}}H_{1,j}+\frac{\partial{L}}{\partial{H_{2,j}}}\nabla_{W^x_{j:}}H_{2,j}$$ Now, $$H_{1, j}$$ affects the loss though $$H_{2,k}\ \forall\ k=1,...,n.$$ So, $$\frac{\partial{L}}{\partial{H_{1,j}}}=\sum_{k=1}^{n}\frac{\partial{L}} {\partial{H_{2,k}}}W^x_{kj}$$ And, $$\frac{\partial{L}}{\partial{H_{2,j}}}=\sum_{k=1}^{n}\frac{\partial{L}} {\partial{Y_{k}}}W^y_{kj}$$ Similarly, $$\nabla_YL$$ can be computed.