You can extract the audio using any youtube downloader, for example 4K youtube downloader and download the audio to lingQ. Then you name the file and generate the transcript. I tried with Chinese and didn't like it much. Besides the many mistakes, it gave traditional characters. Does anybody know if is it possible to select the output for simplified hanzi?

Thank you for the heads-up! The button is located at the bottom of the page (off the screen on my PC) and if you aren't looking for it, and scroll up, you wouldn't even see it!

I just tried it by uploading the next chapter of the 3rd Harry Potter book that I hadn't gotten to yet (ch 9 at 50mins) and it worked perfectly! Even better, since the text is in Iberian Spanish and the audio is in Latin American Spanish, by doing it this way, the text now matches the audio perfectly. As an added bonus, it created the whole chapter complete, without splitting it up into 3 or 4 parts. It took about 15-20 minutes for the whole audio to be processed.

I'm sure the author's guild wouldn't be pleased, but as long as the text isn't shared, I think it's perfectly legal.

In fact, here is a whole list of such tools: https://www.futuretools.io/?tags-n5zn=speech-to-text

I tried it with Hindi and found freesubtitles.ai much better. Conformer-1 also gave the result in Latin alphabet instead of devanagari and had more mistakes.

Yes, for the original whisper:

--output_format srt

[Edit] technically the original whisper already outputs all formats by default when used from the command line

For whisper.cpp:

-osrt

This is fantastic. I am now free (the freesubtitles site is currently overrun by 90+ requests, so basically unusable). Now I can run it myself :)

MacWhisper is fantastic, thanks for posting it.

The original whisper works but is rather slow on macOS, because the underlying PyTorch doesn't take advantage of MPS or the accelerate framework. I assume the situation will improve in the future, see this for more information: https://github.com/pytorch/pytorch/issues/77764#issue-1240333853

I myself run whisper.cpp (https://github.com/ggerganov/whisper.cpp) on a Mac and have no problems. Compilation instructions are in the readme. Happy transcribing!

Sorry to hear that. Among the supported languages Chinese is just at the threshold of being usable and the results certainly cannot be compared to what can be achieved in English, Spanish or even German.

When you say 90%, that is actually a great outcome, even Openai gives us a number of around 85% afaik, and that is on pristine test data. As you say, the vast majority of mistakes are substitutions, but insertions or omissions would be even worse. I myself find the accuracy to be fine for my purposes, although I don't use LingQ for reading transcripts anymore, because, indeed, the inaccuracies combined with LingQ's way of handling Chinese, create a bit of a mess. When I did read transcripts here, I used to disable highlighting to reduce distractions and be able to focus on the text.

I never tried objectively difficult content (just stuff I find difficult to understand) but one suggestion would be to try the large model, if you haven't already, it is really slow and memory intensive but definitely more accurate.

Regarding proof-reading, I don't think we should ascribe Whisper anything resembling intelligence, it just makes predictions - basically like a weatherman it is sometimes right and sometimes wrong. The audio is split into 30 second chunks and they are processed one after the other in sequence, Whisper couldn't go back and correct itself, even if it had any understanding of language. It doesn't really know what word "makes sense" in a given context. Certainly, my knowledge of machine learning is cursory at best but I don't think I'm mischaracterizing its capabilities here.

That being said, I have seen a few instances of emergent capabilities in regards to translations, which is rather fascinating.

My laptop has only intel hd graphics, that is why I mentioned about the cpu only.

Thank you for the link, I will use it from now on since it is faster and optimized :)

I don't really know about the original whisper, but since we discussed whisper.cpp previously here is an idea on how this might work. Although it has to be said I don't know Windows and don't have access to such a machine currently (I'm winging it). So you will have to figure some things out on your own, but you can always ask Google or ChatGPT (e.g. "how do I run FFmpeg on Windows?").

You should be able to just download a Windows executable from the release page on GitHub (https://github.com/ggerganov/whisper.cpp/releases), look under Assets for something like: 'whisper-blas-bin-x64.zip'

Once unzipped you should see 'main.exe' which is the actual transcription software. Of course you need to have a model, the easiest way might be to download one from here: https://huggingface.co/datasets/ggerganov/whisper.cpp/tree/main

I mainly use the medium model for Chinese, which reaches good enough accuracy while running at moderate speed.

The (input) audio has to be in a compatible format as well, I use FFmpeg for the conversion, but there are certainly other ways. You can grab a current Windows binary from here: https://github.com/BtbN/FFmpeg-Builds/releases look for something like 'ffmpeg-master-latest-win64-gpl.zip' or LGPL, but nothing containing "shared" or "Linux".

You wouldn't really install this program like a regular Windows program, instead you can drag-and-drop the exe file in a terminal window (command line, not Powershell), type '-i' drag-and-drop the audio file and then add the following '-ar 16000 -ac 1 -c:a pcm_s16le output.wav'

The end result should look something like this:

\C\Users\Path\to\ffmpeg.exe -i \C\Users\Path\to\input.mp3 -ar 16000 -ac 1 -c:a pcm_s16le output.wav

Now everything should be in place to run whisper; again, by either dropping the 'main.exe' into the terminal window or by typing the path directly. You will have to do the same with the model file.

Here is how it looks for me:

./main -m models/ggml-medium.bin -f /Users/bamboo/test.wav -l zh -osrt

On Windows:

./main would become something like: \C\Users\Path\to\whisper.main

Similarly for the model file \C\Users\Path\to\ggml-medium.bin

And the audio file.

I don't know if this makes any sense. If you are looking for a simpler way, I'm sure many people have already built applications around whisper, that are one-click installable. Maybe look around a bit.

I'm the developer of freesubtitles.ai, glad you like it! The amount of people in the queue has fluctuated quite a bit, right now I'm just paying out of pocket for the servers to be able to let people transcribe content for free, I'm planning to offer some paid features that should hopefully bring in enough revenue whereby I can scale the project more and decrease the queue time. The goal is to have people using the public queue wait no more than 30 minutes (or, hopefully less). So far though people have already transcribed thousands of hours for free (the numbers on the site aren't accurate since I had to move to a new server) so thus far I would consider it a success!