Assume the transformer is trained on 512 max length sentences:

  1. can we fine-tune it on 256 max-length sentences?

  2. If we can fine-tune it, how is it even possible because the input shapes are different, and how is the change happening in high-level overview from weights matrices to end layer

  3. Does every transformer decoder processes one token at the same time, or does it process all tokens at a time if it doesn't, how the decoder process all tokens at the same time?

  4. I have doubt that the process of output is different for Sentiment Analysis and Text generation in Transformer Decoder architecture because in text generation, the decoder process one token at a time, while the sentiment analysis need not be one-token, it can be all tokens at the same time, right? So, how is this difference in both examples, the decoder is able to capture by its architecture, Does all decoder process one token at the same time or all tokens at same time?, If not how is same architecture is able to capture both examples as mentioned above?

  • $\begingroup$ Please, one question per post, even if the questions are in the same context. $\endgroup$
    – nbro
    Apr 22, 2023 at 15:36

2 Answers 2

  1. You can implement the Transformer architecture such that you can indeed change the block size. It can be made smaller and larger. It should be noted, however, that there is no guarantee on performance if you change the max-sequence length. I have no clue how much performance you are going to lose by changing it.
  2. Not too much in the transformer architecture is based on the actual max-sequence length, and all of it can be implemented such that it can be changed. The only things in my own implementation which are based on the block size are the positional embedding (before the actual blocks) and the masking buffer. The positional embedding is done in the original paper using sine/cosine embeddings which can be generalized to any length. The masking buffer can simply be generated on the spot if you would want that. The rest is not based on the actual max-sequence length and is simply done using the size of the batch and matrix multiplications.
  1. The transformer is not a recurrent neural network. It does not need a hidden/carry state from the previous computation to continue with the next computation. It simply needs to context (often called prompt) and does all the calculations based on that context. So the whole context is fed into the network, and out comes the prediction for the next token.
  2. It does process all tokens at the same time.

If you have further questions, im happy to answer them.

  • $\begingroup$ I din't understand few things, 1.) like how a single transformer is able to manage multiple variable input lengths, like after attention mechanism in original transformer paper there is Fully Connected Neural Network right, how is variable length is getting compatible with this FNN? 2.) " The output of each step is fed to the bottom decoder in the next time step, and the decoders bubble up their decoding results just like the encoders did. " This is text from jalammar about transformers, but he mentioned that the transformer process one step at a time $\endgroup$ Apr 22, 2023 at 17:12

I am explicitly elaborating on question 3. I feel the other answers are complete with regard to the other 3 questions posed.

During Inference, the decoder is auto-regressive. I.E. it has to process the next token one at a time.

During Training, the decoder is trained using teacher forcing. Since we already have the ground truth label, we can just pass the golden (provided) tokens {x0,x1....xn}. This permits efficient training.


  • $\begingroup$ So, in the training as you mentioned, decoder processes one token at a time right? Then how is decoder is processing all tokens at the same time $\endgroup$ Apr 22, 2023 at 20:12
  • $\begingroup$ Teacher forcing is a technique that processes all of the ground truth tokens at once. Hence why the decoder input ids are the labels shifted to the right. $\endgroup$
    – Thomas K
    Apr 22, 2023 at 22:17
  • $\begingroup$ I mean "right" in the sense Am I correct? or Am I right?, can you explain how teacher forcing process all tokens at once, because teacher forcing actually works one-token at a time like If the model ground truth is "He is good", Then first we give "He" to model as input and then pass the "is"(As per our ground truth) in next time step, so that means the transformer has to process one token at a time, Am I correct? $\endgroup$ Apr 23, 2023 at 0:25
  • 1
    $\begingroup$ By right, I mean left/right. The labels would be "I have a cat </s>" and the decoder inputs would be "<s>I have a cat". You are correct, teacher forcing works one token at a time. But we can run all steps in parallel during training for the decoder because we already have all the known tokens. $\endgroup$
    – Thomas K
    Apr 23, 2023 at 16:18

Not the answer you're looking for? Browse other questions tagged .