How to play raw PCM data coming through a websocket

[mibrahim2001]

Hi everyone,

I'm working on a conversational chatbot that leverages Deepgram Voice Agent, and I need low-latency, real-time audio streaming. Currently, I'm receiving raw PCM data via a WebSocket, converting it into a Uint8Array, and appending each new chunk to a buffer. Here's my current implementation:

const handleBinaryAudioData = async (data: Blob | ArrayBuffer) => {
    try {
      // Convert blob/buffer to Uint8Array if needed
      const newChunk = new Uint8Array(data);

      // Add the new chunk to our buffer array
      audioBufferRef.current = audioBufferRef.current
        ? new Uint8Array([...audioBufferRef.current, ...newChunk])
        : newChunk;
    } catch (error) {
      console.error("Error processing binary audio chunk:", error);
    }
  };

At the end, I convert this aggregated buffer into a WAV file and play it using Expo AV. However, this approach is not optimal for real-time audio streaming due to latency concerns.

I've been exploring this library since it seems promising for achieving lower latency streaming. Unfortunately, the documentation is still under development, and I haven't been able to figure out how to best integrate it into my workflow.

Questions:

1. How can I stream raw PCM data in real time without aggregating chunks and converting to a WAV file?
2. What's the best way to process continuous PCM data using this library?
3. Are there any examples or best practices for handling low-latency audio streams with this tool?

[michalsek]

Hello 👋

You might want to take a look at our example, which uses openAI api.

Starting in 0.4.2 (which should be published today), re-sampling (done in temporary function goofyResample) shouldn't be necessary, as resampling will be handled by the AudioBufferSourceNode itself.

In order to use raw PCM data with the library, you have to convert the ArrayBuffer to array of float and create an AudioBuffer that will use this array to play a sound with AudioBufferSourceNode. This way you can omit the requirement for aggregating chunks into a WAV file, which is usually the case with audio decodings libs.

For each chunk of the audio data, you have to create separate AudioBufferSourceNode as well as AudioBuffer and use audioContext.currentTime and durations of the buffer to calculate when to play given chunk.

Simplified example:

const aCtx = new AudioContext();
const dataSampleRate = 24000; // Sample rate in which deep gram is sending the PCM chunks.

/* ... */

const playPCMBuffer = async (data: Blob | ArrayBuffer) => {
    try {
      // Convert blob/buffer to Uint8Array if needed
      const newChunk = new Float32Array(data);
      const bufferData = [];

      for (let i = 0; i < newChunk.length; i += 1) {
        bufferData[i] = newChunk[i];
      }
      
      const audioBuffer = aCtx.createBuffer(1, newChunk.length, dataSampleRate);
      audioBuffer.copyToChannel(0, bufferData);
      
      const bufferSource = aCtx.createBufferSource();
      bufferSource.buffer = audioBuffer;
      bufferSource.connect(aCtx.destination);
      bufferSource.play(lastChunkStopTime);
    } catch (error) {
      console.error("Error processing binary audio chunk:", error);
    }
  };

[mibrahim2001]

Hey @michalsek , thank you for the excellent example. I had previously tried a similar approach, but I still encountered brief gaps between audio chunks, which resulted in choppy playback. It seems that a more robust solution would be to implement a real-time streaming channel where audio data is continuously appended and played seamlessly. One promising approach to achieve this is by using the Web Audio API's AudioWorklet. The AudioWorklet API (which has superseded the older ScriptProcessorNode) allows for low-latency, real-time audio processing and is well-suited for dynamically streaming audio data.

[michalsek]

Agree, In near future we would like to explore some [experimental implementation of audio worklet]s(#289) through react-native-worklets-core.

The API will be different than AudioWorkletProcessor offered by web audio, but should be also easier to work with :)