PSA: Realtime audio frontend demo for macOS

[lunixbochs]

This is a working-out-of-the-box demo for realtime speech recognition on macOS with wav2letter++

This is based on my C API in #326
There's a src dir in the w2l_cli tarball with the frontend source (w2l_cli.cpp) and scripts/instructions for building this all from scratch.

to install:

wget https://talonvoice.com/research/w2l_cli.tar.gz
tar -xf w2l_cli.tar.gz && rm w2l_cli.tar.gz
cd w2l_cli
wget https://talonvoice.com/research/epoch186-ls3_14.tar.gz
tar -xf epoch186-ls3_14.tar.gz && rm epoch186-ls3_14.tar.gz

to run:
./bin/w2l emit epoch186-ls3_14/model.bin epoch186-ls3_14/tokens.txt

Then speak, and you should see emissions (letter predictions) in the terminal output after you speak, for example:

$ ./bin/w2l emit epoch186-ls3_14/model.bin epoch186-ls3_14/tokens.txt 
helow|world
this|is|a|test|of|wave|to|leter

Language model decoding is also wired up via ./bin/w2l decode am tokens lm lexicon, but as per #326 it segfaults right now when setting up the Trie.

There are more pretrained english acoustic models at https://talonvoice.com/research/ you can try as well.

[timdoug]

The w2l binary points to dynamic library that isn't present:

$ ./bin/w2l 
dyld: Library not loaded: @rpath/libclang_rt.asan_osx_dynamic.dylib
  Referenced from: .../bin/w2l
  Reason: image not found
Abort trap: 6
$ otool -L ./bin/w2l | grep rpath
	@rpath/libaf.3.dylib (compatibility version 3.0.0, current version 3.6.2)
	@rpath/libmkldnn.0.dylib (compatibility version 0.0.0, current version 0.18.1)
	@rpath/libiomp5.dylib (compatibility version 5.0.0, current version 5.0.0)
$ ls bin/
libaf.3.dylib     libafcpu.3.dylib  libiomp5.dylib    libmkldnn.0.dylib libmklml.dylib    w2l
$

Fixed with install_name_tool:

$ install_name_tool -change @rpath/libclang_rt.asan_osx_dynamic.dylib /Library/Developer/CommandLineTools/usr/lib/clang/10.0.1/lib/darwin/libclang_rt.asan_osx_dynamic.dylib bin/w2l 
$ ./bin/w2l 
Usage: ./bin/w2l emit   <acoustic model> <tokens.txt>
Usage: ./bin/w2l decode <acoustic model> <tokens.txt> <language model> <lexicon>
$

This is on 10.14.5 with the command-line dev tools installed, not full Xcode; the path may be different in that case / different versions / etc.

[lunixbochs]

Yep, I accidentally released the version compiled with -fsanitize-address because I was trying to debug the segfault. I’ll just upload one without that when I’m at my computer next.

[lunixbochs]

Ok, I uploaded a new version at the same URL that wasn't compiled with -fsanitize-address, and with a small improvement to capture more audio at the start of an utterance.

[cogmeta]

will it build successfully on Ubuntu?

[lunixbochs]

No, it uses a MacOS framework for audio capture.

[cogmeta]

as a favor, can you please provide a example with input from a wav file (and does not use any of the mac os audio capturing stuff) and printing the result? I was successfully able to build everything including libw2l.a on ubuntu 16.04 but now stuck with sample example.

[lunixbochs]

No, wav2letter already has code for loading an audio file in the right format using libsndfile in the featurization path, and the Test/Decode binaries can already run against sound files if you put them in the right dataset format.

[cogmeta]

Thanks, i did try ./Decoder but it show no results.

./Decoder -test ./data/ -am ../../../epoch186-ls3_14/model.bin -lm ../../wav2letter/src/decoder/test/lm.arpa -showletters -show
Loading the LM will be faster if you build a binary file.
Reading ../../wav2letter/src/decoder/test/lm.arpa
----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100

---

|T|:
|P|:
|t|: my|internet|is|not|working|man
|p|:
[sample: 0, WER: 100%, LER: 100%, slice WER: 100%, slice LER: 100%, progress: 100%]

[lunixbochs]

I think you should open a new issue for that. It's possible your sound file is in the wrong format. Make sure it's 16 bit 16000khz 1 channel.

[maiasmith]

FYI

wget https://talonvoice.com/research/w2l_cli.tar.gz
...
ERROR: cannot verify talonvoice.com's certificate, issued by ‘CN=Let's Encrypt Authority X3,O=Let's Encrypt,C=US’:
  Issued certificate has expired.

[lunixbochs]

Thanks, the cron job to restart nginx on cert auto renewal wasn’t working. Fixed.

[sapphire-arches]

I hacked together a Linux port of this here. If you don't want to lose your mind trying to get it to build, I recommend using the provided Nix derivations. I had to make several changes to the upstream w2l C API to get it working against the latest wav2letter, which are captured in this fork of wav2letter. It "works" in the sense that it'll pass data to and from the Wav2Letter system, though using epoch186 kindly provided by @lunixbochs results in somewhat underwhelming performance. That might be due to my sketchy normalization, bad microphone, and weak grasp on many of the deep technical details involved in actually deploying an accurate speech to text solution.