4
$\begingroup$

While fine-tuning a decoder only LLM like LLaMA on chat dataset, what kind of padding should one use?

Many papers use Left Padding, but is right padding wrong since transformers gives the following warning if using right padding " A decoder-only architecture is being used, but right-padding was detected! For correct generation results, please set padding_side='left' when initializing the tokenizer."

The attention mask will anyways ignore the padding tokens.

Improve this question
$\endgroup$
2
  • $\begingroup$ Next time, please, put your question in the title. Thanks. $\endgroup$
    – nbro
    Commented Jul 26, 2023 at 10:08
  • $\begingroup$ I'm concerned about this as well, and the generate() method in the transformers library explicitly suggests that decoder-only models should use the left padding method. Is there some correlation? I would also like to know the reason for this left padding. ![enter image description here](i.sstatic.net/Qr63L.png) $\endgroup$
    – 尹雅博
    Commented Jul 29, 2023 at 11:13

2 Answers 2

Reset to default
9
$\begingroup$

I got an answer to this question, probably a correct explanation.

In decoder-only model architectures, the output of the model is a continuation of the model input.

For example, input: I love apple [pad] [pad]. The output of the model will contain the input and add additional output information.

For example, output: I love apple [pad] [pad],because it is delicious.

This would result in [pad] being stuck in the middle of the text. It is very bad for the model to process text. If we use left-padding, the output of this model will be

output : [pad] [pad] i love apple,because it is delicious.

Such complete semantic information is continuous.

ref:https://github.com/huggingface/transformers/issues/18388#issuecomment-1204369688

Improve this answer
$\endgroup$
1
  • $\begingroup$ I don't think this a complete answer. We could simply take the output probabilities correponding to the input "apple" and overwrite the first pad token with the new token. I think the real answer is just that it's slightly more convenient not to have to worry about this by leftpadding everything $\endgroup$
    – johannesack
    Commented Oct 13 at 2:10
1
$\begingroup$

Decoder only text LLMs are autoregressive. Like any autoregressive function if you do not left pad there will be no prior token to your intial token for efficient gradient descent. In most text generation models, the process of token generation is AutoRegressive, which means that the model generates one token at a time, conditioned on the previous tokens it has generated 2.

In the equation you provided, X = (A, b, c) X = F(X(-1)), it seems like X is a vector of length 3, and F is a function that takes the previous value of X as input and returns the next value of X. This is a recursive definition of X, where the current value of X depends on the previous value of X. This is similar to the AutoRegressive process used in text generation models, where the current token depends on the previous tokens generated by the model.

MarvsAI

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .