0
$\begingroup$

Does 1-bit quantization machine learning exist?

Pytorch's docs on "Quantization" define it as:

techniques for performing computations and storing tensors at lower bitwidths than floating point precision.

torch.bool tensors exist with Pytorch, but what's the use for them if most layers use floats? I see the smallest bitwidth besides torch.bool is torch.quint4x2 (unsigned 4-bit integer), but only one sort of layer (nn.EmbeddingBag sparse layer) supports it.

$\endgroup$
2
  • 1
    $\begingroup$ I don't have the time for a longer answer, but essentially no. There is 1.58-bit (ternary) networks, but only for inference (the "forward" pass). You need more bits to do gradient descent (the "backward" pass), otherwise the gradients end up pretty much random. I've tried implementing networks purely with logic gates (e.g. 3-input majority, where one input is the weight), but it couldn't train past a single layer. $\endgroup$ Commented Jun 1 at 13:15
  • $\begingroup$ @programjames The 1-bit BitNet model was the precursor to the 1.58 bit models. If you read The Era of 1-bit LLMs paper, where they talk about 1.58 bit models, they clearly reference the original BitNet paper that introduced 1-bit quantizations. $\endgroup$
    – ahron
    Commented Jun 24 at 6:09

2 Answers 2

1
$\begingroup$

1-bit quantization does exist, at least at the inference stage: a common approach is to constrain weights (and sometimes activations) to be -1 or +1. I'd recommend this survey paper for a good overview.

However, as @programjames mentioned, binarization during training is much more difficult (at least with traditional SGD) as SGD explores through "small and noisy steps", which isn't really possible with such low precision.

During inference, however, binary networks can have some nice properties, like being able to perform convolutions through XNOR and bit-counting operations.

$\endgroup$
2
  • $\begingroup$ I thought Adam was better than SGD for sparse or noisy gradients. $\endgroup$
    – Geremia
    Commented Jun 1 at 22:10
  • $\begingroup$ XNOR-Net seems interesting, as XNOR can be used to find the similarity between bit sequences. $\endgroup$
    – Geremia
    Commented Jun 1 at 22:16
1
$\begingroup$

Yes, 1-bit LLMs are a thing (although, technically, they are 1.58 bit LLMs because they use a ternary system using {-1, 0, +1}). This paper explores practical implementations of 1.58 bit LLMs.

1-bit quantization of LLMs is explored in this 2023 paper about BitNet models. They use BitLinear instead of the standard nn.Linear. This git page has the full implementation.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .