Transcriptions

[pagdot]

I've investigated doing transcriptions for JB shows in the past without reaching a satisfying conclusion yet.

What I'd expect from a decent transcription would be:

• A decent correctness of the result.
  • Specific terminology and names can be hard to get right. Nice to have would be to be able to correct results and feed them back to improve detection
• Speaker diarisation (=recognising who speaks when)
• Detecting sentences/punctuation

A few services I took a look at:

Service/framework	quality	Speaker diarisation	punctuation
Youtube transcription (exported with youtube-dl)	B	❌	❌
IBM Watson	C	✅	❌
SpeechBrain	?	✅	?
DeepSpeech	?	❌	❌
AssemblyAI	B-	✅	✅
Whisper by OpenAI (medium model)	A	❌	✅
pyannote.audio	❌	✅	❌

I tested a combination of Youtube and IBM watson (free tier) in the past: https://gist.github.com/pagdot/3b39187c6e0ca18dedd1f1108338855f

The result was... ok. Not great, but better than nothing.

In my google collab, I further found a test with DeepSpeech by Mozilla

If anyone is interested in also taking a look, Google Colab is great way to test in on a big GPU offered by Google and there often example projects either by the projects themselves or the community for Colab.

Either way a platform to run the transcription on in production would be required and maybe even a way to contribute in their quality.
Could imagine pushing the results in this or another git repository, so that the community can make PRs with fixes

Edit:

2022-08-18: Fixed youtube entry in table (sadly it has no punctuation); added entry for Assembly AI

[gerbrent]

this is really amazing, thanks for all this!! Perhaps dreams do come true.....

[elreydetoda]

This is really cool!

Also, I don't know if it helps at all, but I also listen to this podcast: https://talkpython.fm/friends-of-the-show/sponsors

And they talk about how they use: https://talkpython.fm/assemblyai

Also, for a transcript format @thechangelog has a really cool way they visualize it (but theirs is all done by hand (from my understanding)): https://changelog.com/backstage/24#transcript

And they have all their stuff on GH as well: https://github.com/thechangelog/transcripts/blob/master/backstage/backstage-24.md

They also had an episode about a tool that one of their members made to help with replacing words in bulk: https://changelog.com/backstage/21

[pagdot]

@elreydetoda

And they talk about how they use: https://talkpython.fm/assemblyai

I'll give it a try :) Pricing looks reasonable. I gave it a test drive and documented the results in https://gist.github.com/pagdot/0a5f3037c6ee2d9ba5d5088af8f9a67d

On first glance it looks better than YT + Watson (and has proper punctuation). When taking a deeper look, both have issues at different places. This could maybe improved by combining the result of multiple engines somehow? Would need a lot of work though.

I think if punctuation is important, it does it and YT and Watson don't, else the YT + Watson seem to be a bit better else.

Also, for a transcript format https://github.com/thechangelog has a really cool way they visualize it (but theirs is all done by hand (from my understanding)): https://changelog.com/backstage/24#transcript

Looks nice, although IMO it is very stretched out/takes a lot of space. Personally I'd prefer something more compact :)

[elreydetoda]

Ya, I could see that. I'm curious of they do that to ensure if you go to share it it'll properly show the text for the transcript when someone navigates to it. 🤔

Like this link you can get by clicking on one of their names: https://changelog.com/backstage/24#transcript-22

[pagdot]

could imagine it IRC/text based where the person is on the left of the transcript

[elreydetoda]

Also, changelog has a Practical AI podcast which might have some more solutions for this. I haven't listened to a lot, but it is pretty good and they talk about some pretty cool technical details. They might have an alternative we could use in their back catalog: https://changelog.com/search?q=practical+ai+text

[gerbrent]

Whisper by OpenAI

Robust Speech Recognition via Large-Scale Weak Supervision

https://github.com/openai/whisper

[FlakM]

Hello yall! I've been looking into #26 and wanted to test the search using the transcription. I wanted to post my findings here.

I've previously worked with deep speech from Mozilla but I've found out that the project has been deprecated. Currently, the go to solution seems to be Coqui STT. I've managed to get the full transcription of the example episode (Linux Action News 264) using rust program you can read the resulting transcript here: output.txt

My conclusions:

• generic language model is not good enough - should be definitely fine-tuned to beloved geeky language (companies names, lingo like systemd etc). It seems like the language model is good enough but there is also larger models
• inference on my dell xps 15 (i7) takes 0.7 of the audio length and takes one core so probably i could speed it out by using multiple threads for different recordings
• currently, the example runs the whole file in a single run, before feeding stt model there should be a speaker recognition pipeline to add nice timestamps and then use streaming recognition on separate chunks
• since I think you have mentioned that you have separate tracks for each speaker we should have enough data to train some model to recognize who is speaking and also annotate the transcript (with guest# fallback)

Please let me know if you are interested in investigating it further. Maybe some deep-learning nerd would love to join?

[gerbrent]

quickly: this sounds amazing @FlakM and yes - lets chase this further!

[FlakM]

Over the weekend I've added vad splitting and instructions on how to run the inference. I've also created couple of issues in the repository if someone wants to help.

The biggest problem now is building a dedicated language model (scorer) more suited to the domain. If you know of some good resources with a large amount of easily downloadable text about linux and technology please leave a comment in the issue 👍

BTW the instructions are relatively straightforward for someone who has dealt with python. If someone wants to pick this issue up I would be happy to help! 💯

EDIT:

I've found an amazing library with bindings to whisper API https://github.com/ggerganov/whisper.cpp Somehow I've missed the info that it has been published by open ai not as a web interface. It handles the VAD, language model and punctuation in one go...

There are a couple of versions of models:

1. tiny - works fast 18s/60s audio produces quite good transcription
2. base - slower 33s/60s audio quite comparable transcription
3. medium - sloooow 426s/60s audio but the quality is outstanding transcription

The idea, for now, is to run base on a larger size (all?) of files and see how it works with search ;)

[gerbrent]

These transcriptions are impressively accurate! Nice work! We'll see what we can do to drum up a little more interest in helping here.... very promising!!