Why should one expect the backward pass to take twice as long as the forward pass?

Question

I have seen it stated that, as a rule of thumb, a backward pass in a neural network should take about twice as long as the forward pass. Examples:

• From DeepSpeed's Flops Profiler docs, the profiler:

measures the flops of the forward pass of a module and the flops of the backward pass is estimated as 2 times of that of the forward pass

• Page 7 of Jared Kaplan's Machine Learning notes:

in which it is claimed that the backward pass requires twice the number of matrix-multiplies needed in the forward pass, for a vanilla neural network

.

I unfortunately don't understand the argument made in Kaplan (not sure where the "two" in the "two matrix multiplications per layer he refers to" comes from).

In particular, any such rule would also seem to be very implementation-dependent, depending on whether local gradients are computed and cached during the forward pass, for instance. But I guess there is a standard implementation of backprop that makes this unambiguous.

If anyone can expand on the logic behind this lore or point me towards other references, I would be grateful.

Answer 1

This is a general fact of automatic/algorithmic differentiation. In the forward transport of tangents as well as any backward transport of gradients, each multiplication node splits in two on the derivative level \begin{array}{kcc} \text{node}&:&v=u·w,\\ \text{forward}&:&\dot v = \dot u·w+u·\dot w,\\ \text{backward}&:&\bar u = \bar v·w,~~~\bar w =\bar v·u, \end{array} where $\bar u=\frac{\partial L}{\partial u}$ etc. The evaluation of scalar nodes gives an equally complex operation for the node derivative and an additional scalar multiplication for the forward or backward sweep, \begin{array}{kcc} \text{node}&:&v=\phi(u)\\ \text{forward}&:&\dot v = \phi'(u)·\dot u\\ \text{backward}&:&\bar u = \bar v·\phi'(u), \end{array} which in combination is also a factor of about 2.

The structure of a layered neural network does not change the occurrence of these operations, it only reduces the organizational overhead. Overhead in general will possibly make the backward sweep slower than the forward sweep, as a tape (of some kind) of operations and values has to be constructed and read backwards. In NN this again is greatly simplified, reducing the difference, due to the layered structure already incorporating the tape.

Answer 2

The "two" in the "two matrix multiplications per layer" has nothing to do with any cached value computed in the previous feedforward pass (in fact forward pass doesn’t need any local gradient) where it only needs "one" matrix multiplication per layer denoting the term $Wx$ as shown in the standard input/output vectorial equation $a=\phi(Wx+b)$ in each layer. Thus it may be easier now to understand that for backprop applying automatic differentiation starting from the last layer to calculate derivatives of loss function w.r.t. each weight parameter in the current layer, it needs two matrix multiplication as shown in equation (1.4.6) in your second reference. The first multiplication denotes $\phi’ \times \phi$ where the former is a column vector and the latter is a row vector and its result is a matrix. And the second matrix multiplication occurs inside the calculation of $\phi’$ which is essentially an error factor vector of the input activation vector from its previous hidden layer obtained via output target error vector multiplied with a diagonal matrix of the last layer’s activation function’s derivatives at each node’s respective net input value during the forward pass if you go through the details of backprop.