Motivated by this post wherein one of the comments mentioned the use-case for encoder-decoder LM. I wanted to know when to use prefix-decoder LM? vis a vis encoder-decoder or causal decoder only architectures?
Large-scale language models can be classified into three main types: encoder-decoder, causal decoder and prefix decoder. Each has its advantages and is used in different LLMs for specific purposes.
Encoder-Decoder: Based on the vanilla Transformer model, the encoder-decoder architecture consists of two stacks of Transformer blocks - an encoder and a decoder. The encoder uses stacked multi-headed self-attention layers to encode the input sequence and generate latent representations. The decoder performs cross-attention on these representations and generates the target sequence. This architecture is used in translation software such as Google Translate.
Causal Decoder: The causal decoder architecture incorporates a one-way attention mask, allowing each input token to pay attention only to past tokens and to itself. Input and output tokens are treated in the same way in the decoder. The GPT series, including GPT-1, GPT-2 and GPT-3, are representative language models built on this architecture.
Prefix Decoder : The prefix decoder architecture, also known as the non-causal decoder, modifies the masking mechanism of causal decoders to allow bidirectional attention on prefix tokens and unidirectional attention on generated tokens.
The choice between these architectures depends on the specific task you want to accomplish. For example, if you want to translate text, you can use the encoder-decoder architecture. If you want to generate text autonomously, you can use the causal decoder architecture. Finally, if you want a combination of both, you can use the prefix decoder architecture.