0
$\begingroup$

For conventional 'Neural Networks', the weights simply act as a transformation in highly multi-dimensional space; for a forward pass, the output is always the same since there is no stochastic weighting component in the process.

However, in Transformers (self-attention based encoder-decoder type architecture to be specific) we get different outputs with the same prompts (assuming $T > 0$). This doesn't make sense to me because the set of weights are always static, so the probability distribution produced should be the same; this simple decoding should yield the same output.

However, in practice, we observe that it is not actually the case. Any reasons why?

$\endgroup$
2
$\begingroup$

Language models produce a probability distribution over a set of words. You determine the next word by sampling from this distribution. So, determining the next word is stochastic even though the distribution is the same given the initial prompt.

$\endgroup$
2
  • $\begingroup$ Worth mentioning the role of T in adjusting the distribution for sampling $\endgroup$ Nov 13 '21 at 12:36
  • $\begingroup$ @NeilSlater It's the best parameter ever and has some useful properties. $\endgroup$
    – SpiderRico
    Nov 15 '21 at 3:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.