0
$\begingroup$

While reading about Module in PyTorch, I came across a new data type called half datatype.

half() method when calls on a Module casts all floating-point parameters and buffers to half datatype.

It is a 16-bit floating-point number as mentioned here.

It is mentioned in Wikipedia that

It is intended for storage of floating-point values in applications where higher precision is not essential for performing arithmetic computations.

It implies that the precision of parameters (say, weights for a neural network) is not important in certain applications and hence one can use half datatype while implementing a neural network.

Did any research support the statement that precision, that is the range of values it takes, of weights, is unimportant for certain applications?

$\endgroup$
0

1 Answer 1

0
$\begingroup$

It’s a tradeoff allowing you to fit a larger model into a fixed RAM budget (ie the size of your GPU). Whether this is a good tradeoff is model- and data-specific, but anecdotally I’ve had good luck with it and usually use half precision to good effect (NLP, mostly).

$\endgroup$
1
  • $\begingroup$ Please make a note of the change in the question. $\endgroup$
    – hanugm
    Jan 17 at 8:20

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .