I am struggling to understand the use of the Convolutional Sequence to Sequence (Conv-Seq2Seq) model. The image below is take directly from the paper and is the nearly canonical diagram of the parallel training procedure. After puzzling over it for quite some time, it has come to seem straight forward to me:
- An input sentence of N tokens can be encoded in one step because the input sentence exists prior to the start of training, and therefore the token-wise convolution can be trivially parallelized. (Compare to RNN encoders, which require N steps)
- During training, an output sentence can similarly be parallelized in the decoder because during training, the entire output sentence is known.
- Therefore, during training, the attention function can be fully parallelized in the two dimensional array of dot products shown below
- Finally, during training, the attention is used to weight the input embeddings and encodings, combined with the output training encodings (as such) and the final output assembled.
This is clearly not the case after the network is trained and evaluation input sequences are translated without reference outputs. I understand from various resources (including but not limited to Gehring's conference presentation) that post-training, output sequences are generated token by token in a fashion vaguely similar to earlier architectures, but I cannot find a clear description of that process.
(I speculate that this is because the parallel training routine was so revolutionary at the time, that the focus of the publications was rightly on the training routines.)
Can someone please help me understand the post-training generation algorithm, if possible in terms of the training diagram?
My current non-confident understanding is that the sentence below would be handled something like the following:
- Prime the decoder with a default token string of
<p> <p> <s>, this results (due to convolution) in a single decoder encoding (as such) input into the attention function, and would hopefully generate the single token<'Sie'>as output - Restart the decoder with the token string
<p> <p> <s> <'Sie'>which would generate two inputs into the attention function and hopefully output<'Sie'> <'stimmen'> - Proceed with lengthening input sentences until the final token generated output ends with a
</s>token signifying the end of the sentence.
If that is a correct understanding, can someone confirm it? If close, can someone correct me?
Convolutional Sequence to Sequence Learning, Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, Yann N. Dauphin, 2017
