I have been looking at the source code of the Stanford Alpaca model and I believe that during inference, the whole instruction + response data is fed into the model normally. Then the instruction part of the label is masked with IGNORE_INDEX to prevent gradient calculation on the instruction.

But I believe that in the transformer network, after the attention blocks and before the last head layer, it should be possible to take only the (embedded) tokens corresponding to the response parts from contexts and avoid predicting from the tokens corresponding to the instruction part altogether. This could potentially save computation, especially when the instruction part is long or when training on back-and-forth conversations, similar to interactions with ChatGPT. So my question is:

  1. Is the gradients calculated by these two approaches the same?

I actually tried to ask GPT-4 about this. Initially, he suggested that my approach would lose some information, but later changed his opinion when I asked him further. Would my approach indeed result in a loss of information or any other drawbacks?

  1. Is the saving, if possible, worth the effort of modifying the model's source code?

Since the change would only affect one matrix multiplication, which is already efficiently computed, I'm unsure if it's worth the trouble. Furthermore, I only know a little bitPyTorch and would definitely struggle with implementing modifications to complicated models written in other frameworks like TensorFlow or JAX.

  • $\begingroup$ Hi @Tianchen Zheng, and welcome to AI Stack Exchange! Right now, this post has two separate questions; typically this site requests that only a single question be asked per post. If possible, please edit the post to only have one question. Thank you for posting, and we look forward to seeing you again on this site! $\endgroup$
    – DeepQZero
    Mar 29, 2023 at 15:57


You must log in to answer this question.