4
$\begingroup$

While fine-tuning a decoder only LLM like LLaMA on chat dataset, what kind of padding should one use?

Many papers use Left Padding, but is right padding wrong since transformers gives the following warning if using right padding " A decoder-only architecture is being used, but right-padding was detected! For correct generation results, please set padding_side='left' when initializing the tokenizer."

The attention mask will anyways ignore the padding tokens.

$\endgroup$
2
  • $\begingroup$ Next time, please, put your question in the title. Thanks. $\endgroup$
    – nbro
    Jul 26, 2023 at 10:08
  • $\begingroup$ I'm concerned about this as well, and the generate() method in the transformers library explicitly suggests that decoder-only models should use the left padding method. Is there some correlation? I would also like to know the reason for this left padding. ![enter image description here](i.stack.imgur.com/Qr63L.png) $\endgroup$
    – 尹雅博
    Jul 29, 2023 at 11:13

2 Answers 2

8
$\begingroup$

I got an answer to this question, probably a correct explanation.

In decoder-only model architectures, the output of the model is a continuation of the model input.

For example, input: I love apple [pad] [pad]. The output of the model will contain the input and add additional output information.

For example, output: I love apple [pad] [pad],because it is delicious.

This would result in [pad] being stuck in the middle of the text. It is very bad for the model to process text. If we use left-padding, the output of this model will be

output : [pad] [pad] i love apple,because it is delicious.

Such complete semantic information is continuous.

ref:https://github.com/huggingface/transformers/issues/18388#issuecomment-1204369688

$\endgroup$
1
$\begingroup$

Decoder only text LLMs are autoregressive. Like any autoregressive function if you do not left pad there will be no prior token to your intial token for efficient gradient descent. In most text generation models, the process of token generation is AutoRegressive, which means that the model generates one token at a time, conditioned on the previous tokens it has generated 2.

In the equation you provided, X = (A, b, c) X = F(X(-1)), it seems like X is a vector of length 3, and F is a function that takes the previous value of X as input and returns the next value of X. This is a recursive definition of X, where the current value of X depends on the previous value of X. This is similar to the AutoRegressive process used in text generation models, where the current token depends on the previous tokens generated by the model.

MarvsAI

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .