Instruments

We have compared 4 easy-to-use off-the-shelf instruments for voice activity / audio activity detection with off-the-shelf parameters (we did not hack / rebuild / retrain the tools and just used them as-is):

• A popular python version of the webrtcvad - https://github.com/wiseman/py-webrtcvad;
• Audiotok from this repo;
• Silero-vad from here - https://github.com/snakers4/silero-vad;
• Speech activity detector from pyannote - https://github.com/pyannote/pyannote-audio-hub#speech-activity-detection

Caveats

• Full disclaimer - we are mostly interested in portable production voice detection in phone calls, not just silence detection or dataset preparation where all of these tools will more or less work just fine;
• In our extensive experiments we noticed that WebRTC is actually much better in detecting silence than detecting speech (probably by design). It has a lot of false positives when detecting speech;
• audiotok provides Audio Activity Detection, which probably may just mean detecting silence in layman’s terms;
• silero-vad is geared towards speech detection (as opposed to noise or music);
• A sensible chunk size for our VAD is at least 75-100ms (pauses in speech shorter than 100ms are not very meaningful, but we prefer 150-250ms chunks, see quality comparison here), while audiotok and webrtcvad use 30-50ms chunks (we used default values of 30 ms for webrtcvad and 50 ms for audiotok );
• Maybe we missed something, but using pyannote had a ton of performance caveats, i.e. it required a GPU out of the box and worked very slowly for small files (but quite fast for long files). Also as far as we dug, streaming / online application was not possible with pyannote with standard provided pipeline;

Testing Methodology

Please refer here - https://github.com/snakers4/silero-vad#vad-quality-metrics-methodology

Known limitations:

• Usually speech is very self-correlated (i.e. if one person speaks, he will continue speaking for some time), but our test is extremely hard, because it essentially gives an algorithm only one chance. Essentially we test how well all of algorithms determine the first / last frame of speech without the luxury of being “in the middle” of speech;
• Since we wanted to provide PR curves, it was a bit tricky with py-webrtcvad and pyannote without essentially rebuilding / modifying C++ code for py-webrtcvad and without modifying pipeline code for pyannote. Therefore we had to interpret binary decisions made by these tools into probabilities or just provide one dot on the curve;
• For production usage of silero-vad or to use it with different new domains, you have to set at least some of the params properly. It can be done via provided plotting tool;

Testing Results

Finished tests:

Portability, Speed, Production Limitations

• Looks like originally webrtcvad is written in С++ around 2016, so theoretically it can be ported into many platforms;
• I have inquired in the community, the original VAD seems to have matured and python version is based on 2018 version;
• Looks like audiotok is written in plain python, but I guess the algorithm itself can be ported;
• silero-vad is based on PyTorch and ONNX, so it boasts the same portability options both these frameworks feature (mobile, different backends for ONNX, java and C++ inference APIs, graph conversion from ONNX);
• Looks like pyannote also is built with PyTorch, but it requires GPU out of the box and is extremely slow with short files;

Also we ran a 30-minute audio file via pyannote and silero-vad:

• silero-vad - 20 seconds on CPU;
• pyannote - 12 seconds on 3090;

This is by no means an extensive and full research on the topic, please point out if anything is lacking.