2
$\begingroup$

I am experimenting with Whisper AI sound-to-text (actually I am using the whisper.cpp clone right now) and am trying to optimize the performance but I have discovered something odd - although I use exactly the same setting every time except for the number of threads, the result isn't exactly the same. In other words, the result varies depending on the number of threads.

Is this expected? If so, why does it happen?

Improve this question
$\endgroup$
2
  • $\begingroup$ Can you please provide a link to this Whisper AI? $\endgroup$
    – nbro
    Commented Feb 11, 2023 at 10:26
  • $\begingroup$ @nbro Added links. $\endgroup$
    – d-b
    Commented Feb 11, 2023 at 12:07

1 Answer 1

Reset to default
2
$\begingroup$

Yes, you can see a comment from discussion here:

https://github.com/openai/whisper/discussions/81

This happens when the model is unsure about the output (according to the compression_ratio_threshold and logprob_threshold settings). The most common failure mode is that it falls into a repeat loop, where it likely triggers the compression_ratio_threshold. The default setting tries temperatures 0, 0.2, 0.4, 0.6, 0.8, 1.0 until it gives up, at which it is less likely to be in a repeat loop but is also less likely to be correct.

I've seen also non deterministic results on audio that was clear. Even more than that it generates non related outputs. Using the whisper.ai API you can not play with the parameters but if you are using the open source models you can fine tune it a little bit to perform better.

Improve this answer
$\endgroup$
1
  • $\begingroup$ I realised that if you use two threads it splits the sound into two pieces and lets one thread start from the beginning while the other thread starts from the middle. What happens with the second thread is that, that is different from using only one thread, it is missing linguistic context. That is, if you only use one thread, when you reach the middle of the sound file it has some understanding of the subject of the speech it transcribes which it uses when it interprets the speech, which (hopefully) improves the result. I think that is what happened in my case. $\endgroup$
    – d-b
    Commented Mar 16, 2023 at 13:35

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .