While reading about Module in PyTorch, I came across a new data type called half datatype.

half() method when calls on a Module casts all floating point parameters and buffers to half datatype.

Is it a 16-bit floating point number as mentioned here?

It is mentioned in Wikipedia that

It is intended for storage of floating-point values in applications where higher precision is not essential for performing arithmetic computations.

It implies that the precision of parameters (say, weights for a neural network) is not important and hence one can use half datatype while implementing neural network.

Did any research support the statement that precision, that is range of values it takes, of weights is unimportant?

  • $\begingroup$ I think you misinterpreted that quote. It says "... in applications where...", so it doesn't necessarily imply that precision is not important when training neural networks. Maybe in certain cases it's not that important. So, I suggest that you revise this post to reflect that: maybe you're still interested in the applications where precision would not be important. If you're looking for research work (e.g. research papers), you should probably use the tag reference-request. $\endgroup$
    – nbro
    Jul 21 at 16:28

It’s a tradeoff allowing you to fit a larger model into a fixed RAM budget (ie the size of your GPU). Whether this is a good tradeoff is model- and data-specific, but anecdotally I’ve had good luck with it and usually use half precision to good effect (NLP, mostly).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.