Blog
For a while now Jitsi Meet has been using the RNNoise library to calculate voice audio detection scores for audio input tracks and leveraging those to implement functionality such as talk while muted and noisy mic detection. However, RNnoise also has the capability to denoise audio.
In this article well briefly go through the steps taken to implement noise suppression using RNnoise in Jitsi Meet.
Whats RNNoise anyway?
RNNoise, as the authors describe it, combines classic signal processing with deep learning, but its small and fast, this makes it perfect for real time audio and does a good job at denoising.
Its written in C which allows us to (relatively) easily use it on the Web by compiling it as a WASM module, that combined with a couple of optimizations gets us noise suppression functionality with very little added latency.
Working with Audio Worklets
Previously Jitsi Meet processed audio using ScriptProcessorNode which handles audio samples on the main UI thread. Because the audio track wasnt altered and we simply extracted some information from a copy of the track, performance issues werent apparent. With noise suppression the track gets modified, so latency is noticeable, not to mention that any interference on the main UI thread will impact the audio quality, so we switched to audio worklets.
Audio worklets run in a separate thread from the main UI thread, so samples can be processed without interference. We wont go into the specifics of implementing one as there are plenty of awesome resources on the web such as: thisand this. Our worklet implementation can be found here.
Webpack integration
Even though using an audio worklet looks fairly straightforward there were a couple of bumps along the road.
First off, and probably the most frustrating part was making them work with webpacks dev server.
Long story short, the dev server has some neat features such as hot module replacement and live reloading, these rely on some bootstrap code added to the output JavaScript bundle. The issue here is that audio worklet code runs under the AudioWorkletGlobalScopes context which doesnt know anything about constructs like window, this or self, however the aforementioned boilerplate code makes ample use of them and there doesnt seem to be a way to tell it that the context in which its running is a worklet.
We tried several approaches but the solution that worked for us was to ignore the dev server bootstrap code altogether for the worklets entry point, which can be configured in webpack config as follows:
That took care of the dev server, however production webpack bundling also introduced boilerplate which made use of the forbidden worklet objects, but in this case its easily configurable by specifying the following output options:
At this point we had a working worklet (pun intended) that didnt break our development environment.
WASM in audio worklets.
Next came adding in the RNnoise WASM module. Jitisi uses RNnoise compiled with emscripten (more details in the project: https://github.com/jitsi/rnnoise-wasm). With the default settings the WASM module will load and compile asynchronously, however because the worklet loads without waiting for the resolution of promises we need to make everything synchronous, so we inline the WASM file by passing in -s SINGLE_FILE=1 to emscripten and we also tell it to synchronously compile it with -s WASM_ASYNC_COMPILATION=0. With that in place everything will be loaded and ready to go when audio samples start coming in.
Efficient audio processing.
Audio processing in worklets happens on the process() callback method in the AudioWorkletProcessor implementation at a fixed rate of 128 samples (this cant be configured as with ScriptProcessorNodes), however RNnoise expects 480 samples for each call to its denoise method rnnoise_process_frame.
To make this work we implemented a circular buffer that minimizes copy operations for optimal performance. It works by having both the buffered samples and the ones that have already been denoised on the same Float32Array with a roll over policy. The full implementation can be found here.
To summarize, we keep track of how many audio samples we have buffered, once we have enough of them (480 to be precise) we send a view of that data to RNnoise where it gets denoised in-place (i.e. no additional copies are required). At this point the circular buffer has a denoised part and possibly some residue samples that didnt fit in the initial 480, which will get processed in the next iteration. The process repeats until we reach the end of the circular buffer at which point we simply start from the beginning and overwrite stale samples; we consider them stale because at this point they have already been denoised and sent.
The worklet code gets compiled as a separate .js bundle and lazy loaded as needed.
Use it in JaaS / using the iframe API
If you are a JaaS customer (or are using Jitsi Meet through the iframe API) we have added an API command to turn this on programmatically too! Check it out.
Check it out!
In Jitsi Meet this feature can be activated by simply clicking on the Noise Suppression button.
Since in this case a sound file is probably worth more than 1000 words, here is an audio sample demonstrating the denoising:
Original audio:
Denoised audio:
Your personal meetings team.
Author: Andrei Gavrilescu
Continued here:
Enhanced noise suppression in Jitsi Meet - Jitsi
- Jitsi - Business VoIP Phone Service | OnSIP - April 26th, 2014 [April 26th, 2014]
- Softonic - Jitsi - Download - April 26th, 2014 [April 26th, 2014]
- The Architecture of Open Source Applications: Jitsi - April 26th, 2014 [April 26th, 2014]
- Jitsi - Wikipedia, the free encyclopedia - April 26th, 2014 [April 26th, 2014]
- FLOSS Weekly 293: Jitsi Meet - Video - May 9th, 2014 [May 9th, 2014]
- Jitsi Overview: FLOSS Weekly 293 - Video - May 9th, 2014 [May 9th, 2014]
- NetSecDemo Secure FTP over Jitsi - Video - May 9th, 2014 [May 9th, 2014]
- Eliyah Moore, Barum Jiu Jitsi Vs Cahel Gonzalez, 5 Star Martial Arts - Video - May 16th, 2014 [May 16th, 2014]
- Jitsi (Mac) - Download - Softonic - August 31st, 2014 [August 31st, 2014]
- Using XMPP chat with Jitsi - Video - September 12th, 2014 [September 12th, 2014]
- Jitsi - SIP Softphone - Video - September 12th, 2014 [September 12th, 2014]
- The Luminosity of Free Software Episode 20 - Video - September 13th, 2014 [September 13th, 2014]
- FOSDEM 2014 - Jitsi Videobridge And Webrtc - Video - September 16th, 2014 [September 16th, 2014]
- Just Linux touch screen and Jitsi - Video - September 16th, 2014 [September 16th, 2014]
- Bitcoin and dark wallet could be used by terrorists. So what? - September 24th, 2014 [September 24th, 2014]
- Top messaging apps flat-out flunk EFF's security review - November 5th, 2014 [November 5th, 2014]
- When I say join me in a jitsi meet... - Video - November 7th, 2014 [November 7th, 2014]
- Jitsi Flasms Dogfooding: Using your own imperfect solutions helps improve them, - Video - November 10th, 2014 [November 10th, 2014]
- IETF91 chairs meeting Honolulu Jitsi Meet presentation - Video - November 17th, 2014 [November 17th, 2014]
- IETF91 chairs meeting Honolulu Jitsi Meet questions - Video - November 17th, 2014 [November 17th, 2014]
- jitsi videollamada - Video - December 26th, 2014 [December 26th, 2014]
- Your first OTR text chat with Jitsi - Video - December 31st, 2014 [December 31st, 2014]
- Skype Encrypted Alt. SIP Jitsi Combo - February 7th, 2015 [February 7th, 2015]
- Jitsi Wikipdia - February 7th, 2015 [February 7th, 2015]
- Jitsi for Mac | MacUpdate - Apple Mac OS X Software & Apps ... - February 11th, 2015 [February 11th, 2015]
- Nexi Unified Communication Jitsi Presentazione full - Video - February 19th, 2015 [February 19th, 2015]
- JITSI - Video - April 8th, 2015 [April 8th, 2015]
- Jitsi Configuration and Review - November 21st, 2015 [November 21st, 2015]
- Jitsi: A Multi-Protocol, Cross Platform Compatible Chat, Call ... - November 21st, 2015 [November 21st, 2015]
- Jitsi (Build 3132) - Business VoIP Phone Service | OnSIP - November 21st, 2015 [November 21st, 2015]
- Jitsi | Tiki Suite - March 26th, 2016 [March 26th, 2016]
- Jitsi - - January 28th, 2017 [January 28th, 2017]
- Jitsi for Mac - Download - jitsi.en.softonic.com - February 5th, 2017 [February 5th, 2017]
- Jitsi | Prometheism.net - February 8th, 2017 [February 8th, 2017]
- GitHub - jitsi/jitsi-meet: Jitsi Meet - Secure, Simple and ... - February 12th, 2017 [February 12th, 2017]
- Edward Snowden's New Job: Protecting Reporters From Spies - WIRED - February 14th, 2017 [February 14th, 2017]
- Snowden helping develop tools to protect journalists and whistleblowers - 'to make the game a little more fair' - Press Gazette - February 16th, 2017 [February 16th, 2017]
- Jitsi - Download - February 18th, 2017 [February 18th, 2017]
- Jitsi Meet - Android Apps on Google Play - March 11th, 2017 [March 11th, 2017]
- Jitsi for Mac OS X Secure instant messaging and VoIP - March 12th, 2017 [March 12th, 2017]
- How to Configure and Set-Up Jitsi - Liberty Under Attack - June 6th, 2017 [June 6th, 2017]
- Jitsi PediaView.com | Prometheism.net - June 17th, 2017 [June 17th, 2017]
- Provisioning | Jitsi | Prometheism.net - June 26th, 2017 [June 26th, 2017]
- Provisioning | Jitsi - June 26th, 2017 [June 26th, 2017]
- FAQ | Jitsi | Prometheism.net - June 26th, 2017 [June 26th, 2017]
- Tsirang vegetable vendors commit to selling local chillies - Kuensel, Buhutan's National Newspaper - June 27th, 2017 [June 27th, 2017]
- FAQ | Jitsi | Prometheism.net euvolution.com - July 3rd, 2017 [July 3rd, 2017]
- FAQ | Jitsi | Futurist Transhuman News Blog | Prometheism.net - July 4th, 2017 [July 4th, 2017]
- Jitsi Meet (advanced) Projects | Prometheism.net - July 6th, 2017 [July 6th, 2017]
- FAQ | Jitsi | Prometheism.net euvolution.com | Futurist - July 6th, 2017 [July 6th, 2017]
- Jitsi | Futurist Transhuman News Blog - prometheism.net - July 14th, 2017 [July 14th, 2017]
- Gladstone gold does club proud | Gladstone Observer - Gladstone Observer - July 18th, 2017 [July 18th, 2017]
- Jitsi softphone for Windows OnSIP Support - July 26th, 2017 [July 26th, 2017]
- Jitsi for Mac : Free Download : MacUpdate - August 1st, 2017 [August 1st, 2017]
- OpenFire Jitsi as Skype(desktop sharing) and Temviewer ... - February 9th, 2018 [February 9th, 2018]
- Jitsi Download - softpedia.com - March 29th, 2018 [March 29th, 2018]
- Jitsi Meet (advanced) Projects - June 14th, 2018 [June 14th, 2018]
- Jitsi (Build 3132) - OnSIP - November 9th, 2018 [November 9th, 2018]
- Download Jitsi 2.10.5550 - softpedia.com - November 9th, 2018 [November 9th, 2018]
- Jitsi Community Forum - November 9th, 2018 [November 9th, 2018]
- FAQ | Jitsi - November 23rd, 2018 [November 23rd, 2018]
- Jitsi for Windows - Secure Instant Messaging and VoIP - February 12th, 2019 [February 12th, 2019]
- How to Create Your Own Video Conference Server using Jitsi ... - March 6th, 2019 [March 6th, 2019]
- What is Jitsi? - open source video conferencing API's ... - April 9th, 2019 [April 9th, 2019]
- 8x8 Acquires Jitsi Video Communications Technology From ... - April 15th, 2019 [April 15th, 2019]
- Latest users topics - Jitsi Community Forum - developers & users - April 15th, 2019 [April 15th, 2019]
- Jitsi - Should I Remove It? - April 20th, 2019 [April 20th, 2019]
- Appear.in Vs. Jitsi: Subscription WebRTC Faces Off Against Open-Source VC - VC - VC Daily - November 18th, 2019 [November 18th, 2019]
- Philippines eyes fitting end to best SEA Games performance ever - Sports Interactive Network Philippines - December 12th, 2019 [December 12th, 2019]
- Fox Sports 1 Dials Up VCC for The Herd With Colin Cowherd - TV Technology - December 21st, 2019 [December 21st, 2019]
- Fox Sports 1 Utilized Video Call Center to Extend Reach of Shows Like The Herd with Colin Cowherd - Sports Video Group - December 21st, 2019 [December 21st, 2019]
- GoodFirms Discloses the Most Recommended Software of 2020 to Streamline the Business Work Processes - Yahoo Finance - February 23rd, 2020 [February 23rd, 2020]
- How The Coronavirus Is Forcing Italy To Become A Digital Country, At Last - Forbes - March 16th, 2020 [March 16th, 2020]
- Here's how Zoom is fixing all of those security issues - The Union Journal - April 12th, 2020 [April 12th, 2020]
- Five Zoom alternatives to combat social isolation with video chat - The Star Online - April 12th, 2020 [April 12th, 2020]
- Zoom Has Privacy Issues, Here Are Some Alternatives - CoinDesk - April 12th, 2020 [April 12th, 2020]
- This is Jitsi Meet, the open source video calling application that the Government uses to prepare for press conferences - Explica - April 12th, 2020 [April 12th, 2020]
- Zoom security: I've researched problems with video conferencing for years here's what you need to know - The Conversation UK - May 3rd, 2020 [May 3rd, 2020]
- Group video calls of up to 100 participants, with encryption and noise cancellation - Explica - May 3rd, 2020 [May 3rd, 2020]
- The best Skype alternatives for video calls on your smartphone - AndroidPIT - May 3rd, 2020 [May 3rd, 2020]