0
$\begingroup$

I've been trying to figure out how to compute the number of Flops in backward pass of ResNet. For forward pass, it seems straightforward: apply the conv filters to the input for each layer. But how does one do the Flops counts for gradient computation and update of all weights during the backward pass?

Specifically,

  • how to compute Flops in gradient computations for each layer?

  • what all gradients need to be computed so Flops for each of those can be counted?

  • How many Flops in computation of gradient for Pool, BatchNorm, and Relu layers?

I understand the chain rule for gradient computation, but having a hard time formulating how it'd apply to weight filters in conv layers of ResNet and how many Flops each of those would take. It'd be very useful to get any comments about method to compute total Flops for Backward pass. Thanks

$\endgroup$
  • $\begingroup$ Hi and welcome to AI SE! Please, ask only one question. You're asking multiple questions. I suggest you edit your post to ask a single question and please focus on the smallest problem you need to solve before going to the next one. $\endgroup$ – nbro May 5 at 10:56
  • $\begingroup$ It's actually just 1 question, how to get total op count without coding in backprop for a layer. I didn't get any answer so I thought I should at least lay the whole issue out for anyone to comment. $\endgroup$ – Joe Black May 5 at 13:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.