I have been reading the Deep Learning book by Ian Goodfellow and it mentions in Section 6.5.7 that

The main memory cost of the algorithm is that we need to store the input to the nonlinearity of the hidden layer.

I understand that back-propagation stores the gradients in a similar fashion to dynamic programming so not to recompute them. But I am confused as to why it stores the input as well?


1 Answer 1


The values need to be stored during each batch only, as the forward values are needed to calculate the gradients.

A relevant equation from back propagation is how to derive $\nabla_{z_j^{(k)}}J$, or the gradient of error function $J$ with respect to an individual pre-non-linearity (or logit) value $z$ for neuron $j$ in layer $k$. In the following, $f(z)$ is the non-linearity, and $a_j$ is the output after the non-linearity, and you know $\nabla a_j^{(k)}$ (which you would derive from next layer, or if $k$ is the output layer, from calculating the gradient due to the loss function):

$$\nabla_{z_j^{(k)}}J = \frac{\partial E}{\partial z_j^{(k)}} = \frac{\partial E}{\partial a_j^{(k)}} \frac{\partial a_j^{(k)}}{\partial z_j^{(k)}} = \nabla a_j^{(k)} f'(z_j^{(k)})$$

The value $z_j^{(k)}$ needs to be stored here to evaluate the gradient $\nabla_{z_j^{(k)}}J$. The value of that gradient is then used in further equations for back-propagation in order to calculate the gradients of the weights, which are usually the parameters you want to update.

The storage required is the number of nodes that feed forward, i.e. the total number of neurons in the network (technically also the input values for calculating the first set of gradients, but you will already be storing those, probably for longer), times the size of the batch.

Once each batch or mini-batch has had gradients back-propagated, there is no further use for the node/neuron outputs, and the memory allocation can be re-used for the next batch.

  • $\begingroup$ To make sure I understood correctly: we need to store the forward values because 1) we need it to calculate at least the first gradient going back, and 2) for hidden units (for example, sigmoid) the gradient also contains that value in its expression? If yes, then can you direct me towards some proof intuitive or otherwise of the generalizability of the second point above? $\endgroup$
    – Kunj Mehta
    Mar 23, 2022 at 22:20
  • $\begingroup$ @KunjMehta I added the equation from backprop that uses the value being stored. There are other backprop equations that use the result from that point on. $\endgroup$ Mar 23, 2022 at 22:40
  • $\begingroup$ Understood. Thank you! $\endgroup$
    – Kunj Mehta
    Mar 23, 2022 at 22:54

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .