0
$\begingroup$

I need to transcript some speech files. I've played around with a number of tools and I found that:

  • OpenAI Whisper transcribes speech with absolute accuracy. It's even more accurate than me.

    • However the timestamps it gives are very imprecise: sometimes they are several seconds off.
  • VOSK has a fantastic support for timestamping words: if you crop the audio file at the start and end of any word detected by VOSK, you hear that word precisely.

    • However it's far less accurate in transcription: it often misunderstand speech.

Is there any way to achieve high accuracy in both tasks?

I have tried a few tools that aim at improving Whisper's timestamp accuracy. Namely WhisperX and whisper-timestamped, but neither of them is remotely close to VOSK.


I'm considering trying to combine Whisper and VOSK: maybe I could transcript some speech using both tools, then map each word detected by VOSK on those detected by Whisper, so that I can use VOSK's timestamps for Whisper's words.

However this task seems quite tricky, since many words are going to differ. I wonder whether anyone has attempted this or found better solutions.

$\endgroup$
3
  • $\begingroup$ how can vosk be accurate for timestamps but bad at predicting words? $\endgroup$
    – Alberto
    Commented Sep 7, 2023 at 0:32
  • $\begingroup$ you can take the timestamps of Vosk and the words from Whisper, and should be good to go no? like "vosk tells you the 4th word starts at 4.1s and ends at 5.3s and whisper tells you that the 4th word is blablabla" $\endgroup$
    – Alberto
    Commented Sep 7, 2023 at 0:34
  • $\begingroup$ @Alberto: try Vosk and you'll see. There's no obvious mapping between the words generated by the two tools. Some words are missing, others are extra, whole phrases differ. $\endgroup$ Commented Sep 7, 2023 at 8:02

1 Answer 1

0
$\begingroup$

Curious about what you're building. I needed timestamps to create subtitles and followed this tutorial. https://picovoice.ai/blog/how-to-create-subtitles-for-any-video-with-python/ https://picovoice.ai/docs/leopard/#add-custom-vocabulary

I was happy with the results, because I could add new words and boost them pretty quickly. I couldn't figure out how to customize VOSK though but it could be me. lol

They were gonna add timestamp and diarization features to whisper, but i dont think they did, worth exploring though https://deepgram.com/openai-whisper

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .