My understanding is that masked self-attention is necessary during training of GPT-2, as otherwise it would be able to directly see the correct next output at each iteration. My question is whether the attention mask is necessary, or even possible, during inference. As GPT-2 will only be producing one token at a time, it doesn't make sense to mask out future tokens that haven't been inferred yet.
As a follow up, does this mean that during inference GPT-2 has a "sample" dimension that is always equal to the time iteration count + the prompt length, i.e. during the first iteration, if the prompt was just the "START" token, then only a vector of length 768 (or whatever the embedding size is) will flow through the network?