5
$\begingroup$

LLMs are understood to generate non-deterministic outputs.

Are there LLMs out there that are capable to producing deterministic outputs for any given input given fixed parameters (like e.g temperature)?

I heard that llama.cpp - if run on a CPU instead of a GPU appears to generate deterministic outputs.

$\endgroup$
1
  • 1
    $\begingroup$ When LLMs "generate output" they are sampling from a probability distribution. In principle, it should always be possible to get the same samples from a probability distribution. Read more here: community.openai.com/t/… $\endgroup$
    – Sam
    Commented Dec 6, 2023 at 22:15

2 Answers 2

6
$\begingroup$

Any LLM that exists could easily be modified to be deterministic. At the present, they sample from a probability distribution for the next word. It is a trivial change to make, so that instead of sampling from this distribution, they always pick the word deemed most likely.

This has nothing to do with running on a GPU vs CPU: the non-determinism is not a function of hardware but software.

In many LLMs, this tradeoff is governed by a temperature parameter (it is called temperature for historical reasons) that parameterizes how much randomness will be in the LLM.

Reference: LLM Text Generation: Why is Determinism so Hard to Achieve?

$\endgroup$
4
  • 2
    $\begingroup$ I think OP wants deterministic sampling behavior for any temperature setting (including high temperature). Setting a random seed will do the trick. $\endgroup$
    – Sam
    Commented Dec 7, 2023 at 17:16
  • $\begingroup$ yes - any idea why (to my knowledge) no LLM offers to define a randomness seed to make it deterministic? IMO this would have plenty of useful applications. $\endgroup$
    – user599464
    Commented Dec 8, 2023 at 14:16
  • $\begingroup$ I don't think this is correct. You can set temperature to 0 and even set a seed (if available), thereby always selecting the most likely next token, but that won't get you the same output from GPT or Claude. $\endgroup$
    – zcahgg1
    Commented 2 days ago
  • $\begingroup$ Here's a link to some discussions on HN news.ycombinator.com/item?id=37446724 $\endgroup$
    – zcahgg1
    Commented 2 days ago
0
$\begingroup$

To answer your question "No there arent any".

Executing an LLM such as llama.cpp on a CPU may indeed lead to more predictable outcomes due to the way it handles floating-point calculations. To see the difference for yourself, simply obtain a copy of llama.cpp and adjust the temperature and seed parameters as needed, or apply both if relevant, and conduct your tests. The same experiment can be replicated with a model running on a GPU to compare results.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .