Adaptive audio stream with latency compensation
A latency compensating adaptive jitter buffer method is implemented in an audio client running on a user device or in an audio server to adaptively adjust the size of a jitter buffer to optimize latency while minimizing packet loss during audio signal transmission. In some embodiments, the jitter buffer is kept to a nominal size for low latency. In response to a queuing delay event being detected, audio production is temporarily stopped and the size of the jitter buffer is temporarily increased to receive all incoming audio packets up to a certain refill level. The method then resumes audio production using accelerated playback to reduce the jitter buffer size back to the nominal size.
1. A method in a receive process of an audio client being executed on a user device or an audio conferencing server for receiving audio data packets, the method comprising:
setting a jitter buffer to a nominal size for low latency;
receiving incoming audio data packets;
storing the audio data packets into the jitter buffer;
producing audio playback using the audio data packets from the jitter buffer at a nominal playback rate;
monitoring the incoming audio data packets for a queuing delay event;
in response to a queuing delay event being detected, stopping production of audio playback;
increasing the size of the jitter buffer to greater than the nominal size to receive and store incoming audio data packets;
storing audio data packets in the jitter buffer up to a refill level greater than the nominal size;
in response to the jitter buffer having stored therein audio data packets reaching the refill level, resuming production of audio playback using the audio data packets from the jitter buffer at an accelerated playback rate faster than the nominal playback rate; and
in response to the jitter buffer size returning to the nominal size, continuing to receive incoming audio data packets and producing audio playback of the audio data packets from the jitter buffer at the nominal playback rate.
2. The method of claim 1 , wherein setting a jitter buffer to a nominal size for low latency comprises:
setting the jitter buffer to the nominal size as determined by the history of the jitter buffer size and a target jitter buffer size for absorbing packet queuing delay variations.
3. The method of claim 1 , wherein monitoring the incoming audio data packets for a queuing delay event comprises:
monitoring the arrival of incoming audio data packets after the jitter buffer has become empty as indication of a queuing delay event.
4. The method of claim 1 , wherein resuming production of audio playback using the audio data packets from the jitter buffer at an accelerated playback rate faster than the nominal playback rate comprises:
resuming production of audio playback using the audio data packets from the jitter buffer at an accelerated playback rate using an audio playback acceleration technique selected from sample rate conversion, asynchronous granular overlap add, pitch synchronous overlap add (PSOLA), phase vocoder, and sinusoidal modeling.
5. The method of claim 1 , wherein resuming production of audio playback using the audio data packets from the jitter buffer at an accelerated playback rate faster than the nominal playback rate comprises:
resuming production of audio playback using the audio data packets from the jitter buffer at an accelerated playback rate by discarding accumulated audio data packets having a voice activity under a given amplitude threshold.
6. The method of claim 1 , wherein resuming production of audio playback using the audio data packets from the jitter buffer at an accelerated playback rate faster than the nominal playback rate comprises:
resuming production of audio playback using the audio data packets from the jitter buffer at an accelerated playback rate by discarding accumulated audio data packets that are silent.
7. The method of claim 1 , wherein monitoring the incoming audio data packets for a queuing delay event comprises:
determining the jitter buffer has become empty;
monitoring the arrival of incoming audio data packets after the jitter buffer has become empty as indication of a queuing delay event; and
measuring a first time duration of the jitter buffer becoming empty until an audio data packet has arrived.
8. The method of claim 7 , wherein storing audio data packets in the jitter buffer up to a refill level greater than the nominal size comprises:
storing audio data packets in the jitter buffer up to a refill level corresponding to a buffer duration equal to the first time duration multiplied by a refill factor, the refill factor being 1 or greater.
9. A cloud-based audio conferencing system for processing audio signals of a conference call, comprising:
a processor configured to execute a receive process for receiving audio data packets, the receive process being configured to:
set a jitter buffer to a nominal size for low latency;
receive incoming audio data packets;
store the audio data packets into the jitter buffer;
produce audio playback using the audio data packets from the jitter buffer at a nominal playback rate;
monitor the incoming audio data packets for a queuing delay event;
in response to a queuing delay event being detected, stop production of audio playback;
increase the size of the jitter buffer to greater than the nominal size to receive and store incoming audio data packets;
store audio data packets in the jitter buffer up to a refill level greater than the nominal size;
in response to the jitter buffer having stored therein audio data packets reaching the refill level, resume production of audio playback using the audio data packets from the jitter buffer at an accelerated playback rate faster than the nominal playback rate; and
in response to the jitter buffer size returning to the nominal size, continue to receive incoming audio data packets and producing audio playback of the audio data packets from the jitter buffer at the nominal playback rate; and
a memory coupled to the processor and configured to provide the processor with instructions.
10. The system of claim 9 , wherein the processor is configured to set the jitter buffer to the nominal size as determined by the history of the jitter buffer size and a target jitter buffer size for absorbing packet queuing delay variations.
11. The system of claim 9 , wherein the processor is configured to monitor the arrival of incoming audio data packets after the jitter buffer has become empty as indication of a queuing delay event.
12. The system of claim 9 , wherein the processor is configured to resume production of audio playback using the audio data packets from the jitter buffer at an accelerated playback rate by discarding accumulated audio data packets having a voice activity under a given amplitude threshold.
13. The system of claim 9 , wherein the processor is configured to:
determine the jitter buffer has become empty;
monitor the arrival of incoming audio data packets after the jitter buffer has become empty as indication of a queuing delay event; and
measure a first time duration of the jitter buffer becoming empty until an audio data packet has arrived.
14. The system of claim 13 , wherein the processor is configured to store audio data packets in the jitter buffer up to a refill level corresponding to a buffer duration equal to the first time duration multiplied by a refill factor, the refill factor being 1 or greater.
15. A device for processing audio signals in an audio communication over a data network, comprising:
a processor configured to execute a receive process for receiving audio data packets, the receive process being configured to:
set a jitter buffer to a nominal size for low latency;
receive incoming audio data packets;
store the audio data packets into the jitter buffer;
produce audio playback using the audio data packets from the jitter buffer at a nominal playback rate;
monitor the incoming audio data packets for a queuing delay event;
in response to a queuing delay event being detected, stop production of audio playback;
increase the size of the jitter buffer to greater than the nominal size to receive and store incoming audio data packets;
store audio data packets in the jitter buffer up to a refill level greater than the nominal size;
in response to the jitter buffer having stored therein audio data packets reaching the refill level, resume production of audio playback using the audio data packets from the jitter buffer at an accelerated playback rate faster than the nominal playback rate; and
in response to the jitter buffer size returning to the nominal size, continue to receive incoming audio data packets and producing audio playback of the audio data packets from the jitter buffer at the nominal playback rate; and
a memory coupled to the processor and configured to provide the processor with instructions.
16. The device of claim 15 , wherein the processor is configured to set the jitter buffer to the nominal size as determined by the history of the jitter buffer size and a target jitter buffer size for absorbing packet queuing delay variations.
17. The device of claim 15 , wherein the processor is configured to monitor the arrival of incoming audio data packets after the jitter buffer has become empty as indication of a queuing delay event.
18. The device of claim 15 , wherein the processor is configured to resume production of audio playback using the audio data packets from the jitter buffer at an accelerated playback rate by discarding accumulated audio data packets having a voice activity under a given amplitude threshold.
19. The device of claim 15 , wherein the processor is configured to:
determine the jitter buffer has become empty;
monitor the arrival of incoming audio data packets after the jitter buffer has become empty as indication of a queuing delay event; and
measure a first time duration of the jitter buffer becoming empty until an audio data packet has arrived.
20. The device of claim 19 , wherein the processor is configured to store audio data packets in the jitter buffer up to a refill level corresponding to a buffer duration equal to the first time duration multiplied by a refill factor, the refill factor being 1 or greater.