I have seen it stated that, as a rule of thumb, a backward pass in a neural network should take about twice as long as the forward pass. Examples:
- From DeepSpeed's Flops Profiler docs, the profiler:
measures the flops of the forward pass of a module and the flops of the backward pass is estimated as
2times of that of the forward pass
- Page 7 of Jared Kaplan's Machine Learning notes:
in which it is claimed that the backward pass requires twice the number of matrix-multiplies needed in the forward pass, for a vanilla neural network
I unfortunately don't understand the argument made in Kaplan (not sure where the "two" in the "two matrix multiplications per layer he refers to" comes from).
In particular, any such rule would also seem to be very implementation-dependent, depending on whether local gradients are computed and cached during the forward pass, for instance. But I guess there is a standard implementation of backprop that makes this unambiguous.
If anyone can expand on the logic behind this lore or point me towards other references, I would be grateful.