0
$\begingroup$

Is it possible to give a large-language-model access to the time it takes to give it's answers and ask it to optimize on that?

An example prompt would be:

From now on, please measure the time it takes for you to give me an answer and optimize that time so that it stays under 500 milliseconds. It is alright if you give me incorrect information because of this constraint.

We can assume that the human brain optimizes for processing speed to lower it's energy usage. How can this be applied to a LLM?

Improve this question
$\endgroup$
2
  • $\begingroup$ if you add "considering that on average every word you process takes X ms" it will probably already work, because you are pretty much just telling it to produce responses of length < 500/X $\endgroup$
    – Alberto
    Commented Dec 16, 2023 at 23:11
  • $\begingroup$ Yes, that would be limiting the output. But is the time needed for an LLMs to answer really linearly depending on the number of inputs tokens and output tokens. Or are there some loops. Kind of how humans are able to reason, arrive at a conclusion, see that the have an edge case and spend more time thinking about the edgecase? My request would be for the LLMs to output "there's an edge case I would need to analyze deeper if I had more time." $\endgroup$
    – Fabian Zeindl
    Commented Dec 17, 2023 at 6:33

1 Answer 1

Reset to default
1
$\begingroup$

It is not currently possible to directly give a large language model access to measure the time it takes to generate its answers and optimize on that.

The model's response time is largely determined by the underlying hardware and software infrastructure, and optimizing it would require significant changes at the system level. While the human brain does optimize for processing speed to lower energy usage, applying this directly to a large language model would require a different approach due to the fundamental differences in their architecture and operation.

However, ongoing research in the field of artificial intelligence continues to explore ways to improve the efficiency and speed of language models.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ Off topic, but this answer seem to be generated by an LLM. Could you confirm or deny that? And could you elaborate on the details of "require a different approach due to the fundamental differences in their architecture and operation.". Which approach would it require? $\endgroup$
    – Fabian Zeindl
    Commented Dec 17, 2023 at 6:35

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .