0
$\begingroup$

For conventional 'Neural Networks', the weights simply act as a transformation in highly multi-dimensional space; for a forward pass, the output is always the same since there is no stochastic weighting component in the process.

However, in Transformers (self-attention based encoder-decoder type architecture to be specific) we get different outputs with the same prompts (assuming $T > 0$). This doesn't make sense to me because the set of weights are always static, so the probability distribution produced should be the same; this simple decoding should yield the same output.

However, in practice, we observe that it is not actually the case. Any reasons why?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

Language models produce a probability distribution over a set of words. You determine the next word by sampling from this distribution. So, determining the next word is stochastic even though the distribution is the same given the initial prompt.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Worth mentioning the role of T in adjusting the distribution for sampling $\endgroup$
    – Neil Slater
    Nov 13, 2021 at 12:36
  • $\begingroup$ @NeilSlater It's the best parameter ever and has some useful properties. $\endgroup$
    – SpiderRico
    Nov 15, 2021 at 3:28

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .