Can you confirm that the transformer works strictly deterministically and there is no randomness inside or between the attention layers?

Question

On a high-level temperature and randomness affect the output of a generative language model:

1. Lower temperature: Produces more focused, conservative, and consistent responses.

2. Moderate temperature: Strikes a balance between creativity and consistency. This setting can be useful for general content generation, where a blend of accuracy and inventiveness is desired.

3. Higher temperature: Generates more creative, diverse, and unexpected outputs.

What I'm not sure of is where exactly randomness (controlled by the temperature) comes into play. I believe to have understood that it's only after a transformer has done its deterministic work, suggesting some probable next words.

Can you confirm that the transformer works strictly deterministically and there is no randomness inside or between the attention layers?

Answer 1

Can you confirm that the transformer works strictly deterministically and there is no randomness inside or between the attention layers?

Of course, there is no injected randomness in a regular Transformer model: you have just and encoder-decoder architecture, with positional encoding, multiple multi-head attention blocks and layer normalization.

The randomness you describe is typical of GPT model family which are generative, being trained to predict the next word token. Basically, you have a distribution over next tokens and you can control the amount of randomness in the sampling by dividing the logits by a temperature $\tau$: in the limit if $\tau\to\infty$ the predicted distribution becames uniform, otherwise if $\tau\to 0$ it exactly resembles a one-hot encoding (so a $1$ in a given place, and all zeros elsewhere.)

I'm not sure of is where exactly randomness comes into play.

For example, at the end of the Transformer you have a dense layer which outputs $K$ logits: unnormalized probabilities over $K$ tokens. You can define a Categorical distribution from the logits $\alpha$ (well, you can also define it from normalized probabilities by applying a softmax on top) from which you can sample the "index" of the next token. To control the sampling you introduce the $\tau$ as follows $\text{Categorical}(\alpha / \tau)$. To generate new text you sample from that: say you start from a context $C$, you forward the model to get the logits $\alpha$ and sample a new token $t$, then update the context $C=C\cup \{t\}$ (so that the model also considers the novel predicted word), and repeat the process again.

Answer 2

That's right, the Attention Layer output is totally deterministic.

The temperature parameter is related to generative tasks (note that this is not the only thing you can do with Attention and the transformers architecture), and it controls how, given the logits (a vector of the same length as the vocabulary $V$, the model samples one token among all the available tokens.

Suppose, for simplicity, that we are doing Causal generation and we have a vector of logits $u$ of length $V$

If you do not sample and take just the most probable token at each time step (i.e the $\arg\max$ of the logits), you always end up with the same generation, given a fixed input.

Otherwise, we can model the distribution of probability over all tokens with a softmax and sample one token from the distribution at each step $t$.

$$\mathbb{P}(x=v_l | x_{1:t-1}) = \frac{\exp(u_l )}{\sum_{l'}\exp(u'_l)} \qquad \text{where}\; v_l = x_{1:t-1}l \;, \forall l \in V$$

The temperature modifies this distribution by warping the previous distribution, multiplying the terms inside the exponential by a factor $t$.

$$\mathbb{P}(x=v_l | x_{1:t-1}) = \frac{\exp(u_l/t)}{\sum_{l'}\exp(u'_l/t)} \qquad \text{where}\; v_l = x_{1:t-1}l \;, \forall l \in V$$

For $t \rightarrow 1$, the distribution gets sharper, as it tends to a indicator function over the maximum token (the $\arg \max$).

For $t \rightarrow 1$ it is equivalent to the softmax formula above, with values greater than 1 it flattens the distribution, i.e the distribution tends to a uniform distribution as $t \rightarrow \infty$

References: https://huggingface.co/blog/how-to-generate

Answer 3

The other answers might be correct for the mathematical abstraction of a GPT-like model, but are false for the actual real-world models. On-device forward passes through any big quantized neural network, including state-of-the-art OpenAI models in 2023, are not deterministic. The difference is not large, but in gpt-4-0314 with temperature 0, you can get a different token sampled every 100 tokens or so.

Edit (August 2023): What I wrote above is partly correct. The hardware issues explain some of the randomness; however, OpenAI's Chat Completions API models have more variance than the old Completions API models, which is some evidence that randomness in GPT-4 is not explained by hardware issues only.

The best guess I've seen is that mixture-of-experts routers operate on batches of queries, and are deterministic (up to hardware issues) on the batch level, not on single queries. There is no information leakage between inputs from different users, but the expert choice in each layer depends on what is in the batch, influencing the output.

Of course, this is speculation, and although the MoE theory seems plausible, it could turn out that GPT-4 inference requires a GPU setup which makes inference less deterministic than on classical dense models.