PyTorch has new functionality torch.inference_mode as of v1.9 which is "analogous to torch.no_grad... Code run under this mode gets better performance by disabling view tracking and version counter bumps."

If I am just evaluating my model at test time (i.e. not training), is there any situation where torch.no_grad is preferable to torch.inference_mode? I plan to replace every instance of the former with the latter, and I expect to use runtime errors as a guardrail (i.e. I trust that any issue would reveal itself as a runtime error, and if it doesn't surface as a runtime error then I assume it is indeed preferable to use torch.inference_mode).

More details on why inference mode was developed are mentioned in the PyTorch Developer Podcast.

I asked a similar question elsewhere but it was the wrong forum.

  • 6
    $\begingroup$ I think the pytorch forums would be the best for this question: discuss.pytorch.org $\endgroup$
    – SpiderRico
    Oct 12 at 20:30
  • $\begingroup$ Hello. Welcome to Artificial Intelligence Stack Exchange :) I haven't immediately closed your post when I first came across it, but it's off-topic for our site. Questions about specific libraries, how to use them, etc., are off-topic here. We focus on the theoretical, social and philosophical aspects of AI. This type of question is more appropriate for Stack Overflow (although you state that it was the wrong forum), or, as suggested in the previous comment, for the specific forum of the library you're using. $\endgroup$
    – nbro
    Oct 13 at 21:40
  • $\begingroup$ Thanks @nbro. I will keep this in mind for the future. $\endgroup$
    – efthimio
    Oct 13 at 21:42

This was recently answered on the PyTorch Forums.

Yes, torch.inference_mode is indeed preferable to torch.no_grad in all situations where inference mode does not throw a runtime error.

The reason it took until version 1.9 to be implemented was precisely because it was originally difficult to ensure that all "unsafe" operations in inference mode were detected and thrown rather than silently ignored, which would have caused confusing results for the user. The torch team did now want to allow users the performance improvement without holding them accountable to the contract they must abide by to attain the improvement. As it is implemented now, anything unsafe in inference mode (or unsafe from using "inference tensors" outside the context of inference mode) should throw an error.

Edward Yang from the PyTorch Developer Podcast says that it has lead to 5% to 10% gains in performance in production at Facebook.


Not the answer you're looking for? Browse other questions tagged or ask your own question.