Method and apparatus for reducing rendering latency for audio streaming applications using internet protocol communications networks
A method and apparatus for reducing rendering latency in a terminal device which receives audio data from a communication network such as, for example, Voice over Internet Protocol (VoIP) communications networks. Received packets are advantageously decoded “immediately” upon receipt, and the decoded data is placed directly in the rendering buffer at a location corresponding to the time appropriate for rendering, without using any intermediate buffer. Then, in accordance with the principles of the present invention and more particularly in accordance with certain illustrative embodiments thereof, packet loss concealment (PLC) routines are advantageously applied preemptively, without first determining whether or not any subsequent packets have or have not been received by any particular time.
1. A method performed by a terminal device for receiving audio data communicated thereto across an Internet Protocol (IP) communications network, the terminal device comprising a packet receiver, a processor and a memory, the memory including a rendering buffer implemented therein, the method comprising:
receiving an encoded audio signal packet using the packet receiver, the encoded audio signal packet being comprised in a sequence of encoded audio signal packets, the sequence of encoded audio signal packets comprising data representative of an audio signal;
in response to receiving said encoded audio signal packet, using a decoder implemented on the processor to decode the received encoded audio signal packet to generate corresponding decoded audio data and storing said corresponding decoded audio data in the rendering buffer;
in further response to receiving said encoded audio signal packet, and without making any determination that one or more encoded audio signal packets in said sequence of encoded audio signal packets subsequent to said encoded audio signal packet have not been received, using the decoder to perform packet loss concealment, thereby generating one or more concealment packets of audio data representative of audio data subsequent to said decoded audio data, and further storing said one or more generated concealment packets of audio data in said rendering buffer;
rendering said decoded audio data stored in said rendering buffer and said one or more generated concealment packets of audio data stored in said rendering buffer as audio output from said terminal device;
saving in said memory a corresponding state of the decoder prior to said generation of each of said concealment packets of audio data;
receiving a previous encoded audio signal packet using the packet receiver, said previous encoded audio signal packet being received after the receipt of the encoded audio signal packet, the previous encoded audio signal packet being comprised in the sequence of encoded audio signal packets prior to said encoded audio signal packet and being representative of audio data for which one of said concealment packets of audio data has been generated and stored in said rendering buffer;
restoring said saved state of said decoder corresponding to said generation of said concealment packet of audio data representative of said audio data represented by said previous encoded audio signal packet;
in response to receiving said previous encoded audio signal packet, using the decoder implemented on the processor to decode the received previous encoded audio signal packet to generate corresponding decoded audio data and storing said corresponding decoded audio data in the rendering buffer so as to replace said concealment packet of audio data representative of said audio data represented by said previous encoded audio signal packet which had been previously stored in said rendering buffer; and
retrieving said received encoded audio signal packet from said holding buffer and using the decoder implemented on the processor to re-decode the received encoded audio signal packet to generate corresponding decoded audio data and storing said corresponding decoded audio data in the rendering buffer;
wherein the memory further comprises a holding buffer, the method further comprising storing said received encoded audio signal packet in said holding buffer.
2. The method of claim 1 wherein audio data comprises speech and wherein the Internet Protocol communications network comprises a Voice over Internet Protocol (VoIP) communications network.
3. The method of claim 1 further comprising:
receiving a subsequent encoded audio signal packet using the packet receiver, said subsequent encoded audio signal packet being received after the receipt of the encoded audio signal packet, the subsequent encoded audio signal packet being comprised in the sequence of encoded audio signal packets subsequent to said encoded audio signal packet and being representative of audio data for which one of said concealment packets of audio data has been generated and stored in said rendering buffer;
restoring said saved state of said decoder corresponding to said generation of said concealment packet of audio data representative of said audio data represented by said subsequent encoded audio signal packet; and
in response to receiving said subsequent encoded audio signal packet, using the decoder implemented on the processor to decode the received subsequent encoded audio signal packet to generate corresponding decoded audio data and storing said corresponding decoded audio data in the rendering buffer so as to replace said concealment packet of audio data representative of said audio data represented by said subsequent encoded audio signal packet which had been previously stored in said rendering buffer.
4. The method of claim 3 further comprising, in further response to receiving said subsequent encoded audio signal packet, and without making any determination that one or more encoded audio signal packets in said sequence of encoded audio signal packets subsequent to said subsequent encoded audio signal packet have not been received, using the decoder to perform packet loss concealment to generate one or more concealment packets of audio data representative of audio data subsequent to said decoded audio data corresponding to said subsequent encoded audio signal packet, and further storing said one or more generated concealment packets of audio data in said rendering buffer.
5. A communications network terminal device for receiving audio data communicated thereto across an Internet Protocol (IP) communications network, the terminal device comprising:
a packet receiver which receives an encoded audio signal packet, the encoded audio signal packet being comprised in a sequence of encoded audio signal packets, the sequence of encoded audio signal packets comprising data representative of an audio signal;
a processor having a decoder implemented thereon; and
a memory, the memory including a rendering buffer implemented therein,
wherein in response to the packet receiver receiving said encoded audio signal packet, the decoder implemented on the processor decodes the received encoded audio signal packet to generate corresponding decoded audio data and stores said corresponding decoded audio data in the rendering buffer, wherein
in further response to the packet receiver receiving said encoded audio signal packet, and without making any determination that one or more encoded audio signal packets in said sequence of encoded audio signal packets subsequent to said encoded audio signal packet have not been received, the decoder implemented on the processor performs packet loss concealment, thereby generating one or more concealment packets of audio data representative of audio data subsequent to said decoded audio data, and further stores said one or more generated concealment packets of audio data in said rendering buffer, and wherein
said decoded audio data stored in said rendering buffer and said one or more generated concealment packets of audio data stored in said rendering buffer are rendered as audio output from said terminal device; and wherein
the packet receiver further receives a previous encoded audio signal packet, said previous encoded audio signal packet being received after the receipt of the encoded audio signal packet, the previous encoded audio signal packet being comprised in the sequence of encoded audio signal packets prior to said encoded audio signal packet and being representative of audio data for which one of said concealment packets of audio data has been generated and stored in said rendering buffer, and wherein
the processor further restores said saved state of said decoder corresponding to said generation of said concealment packet of audio data representative of said audio data represented by said previous encoded audio signal packet, and wherein
in response to receiving said previous encoded audio signal packet, the processor uses the decoder implemented thereon to decode the received previous encoded audio signal packet to generate corresponding decoded audio data and store said corresponding decoded audio data in the rendering buffer so as to replace said concealment packet of audio data representative of said audio data represented by said previous encoded audio signal packet which had been previously stored in said rendering buffer, and wherein
the processor further retrieves said received encoded audio signal packet from said holding buffer and uses the decoder implemented thereon to re-decode the received encoded audio signal packet to generate corresponding decoded audio data and store said corresponding decoded audio data in the rendering buffer;
wherein said processor further saves in said memory a corresponding state of the decoder prior to said generation of each of said concealment packets of audio data; and
wherein the memory further comprises a holding buffer, and wherein said processor further stores said received encoded audio signal packet in said holding buffer.
6. The terminal device of claim 5 wherein said audio data comprises speech and wherein the Internet Protocol communications network comprises a Voice over Internet Protocol (VoIP) communications network.
7. The terminal device of claim 5 wherein the packet receiver further receives a subsequent encoded audio signal packet, said subsequent encoded audio signal packet being received after the receipt of the encoded audio signal packet, the subsequent encoded audio signal packet being comprised in the sequence of encoded audio signal packets subsequent to said encoded audio signal packet and being representative of audio data for which one of said concealment packets of audio data has been generated and stored in said rendering buffer, wherein
the processor further restores said saved state of said decoder corresponding to said generation of said concealment packet of audio data representative of said audio data represented by said subsequent encoded audio signal packet, and wherein
in response to receiving said subsequent encoded audio signal packet, the decoder implemented on the processor further decodes the received subsequent encoded audio signal packet to generate corresponding decoded audio data and stores said corresponding decoded audio data in the rendering buffer so as to replace said concealment packet of audio data representative of said audio data represented by said subsequent encoded audio signal packet which had been previously stored in said rendering buffer.
8. The terminal device of claim 7 wherein, in further response to receiving said subsequent encoded audio signal packet, and without making any determination that one or more encoded audio signal packets in said sequence of encoded audio signal packets subsequent to said subsequent encoded audio signal packet have not been received, the decoder implemented on the processor further performs packet loss concealment to generate one or more concealment packets of audio data representative of audio data subsequent to said decoded audio data corresponding to said subsequent encoded audio signal packet, and further stores said one or more generated concealment packets of audio data in said rendering buffer.
9. A non-transitory tangible computer readable medium product comprising non-transitory data recorded thereupon, the data recorded thereupon comprising program instructions which when executed by a computer system implement a method performed by a terminal device for receiving audio data communicated thereto across an Internet Protocol (IP) communications network, the terminal device comprising a packet receiver, a processor and a memory, the memory including a rendering buffer implemented therein, the method comprising program instructions for:
receiving an encoded audio signal packet using the packet receiver, the encoded audio signal packet being comprised in a sequence of encoded audio signal packets, the sequence of encoded audio signal packets comprising data representative of an audio signal;
in response to receiving said encoded audio signal packet, using a decoder implemented on the processor to decode the received encoded audio signal packet to generate corresponding decoded audio data and storing said corresponding decoded audio data in the rendering buffer;
in further response to receiving said encoded audio signal packet, and without making any determination that one or more encoded audio signal packets in said sequence of encoded audio signal packets subsequent to said encoded audio signal packet have not been received, using the decoder to perform packet loss concealment, thereby generating one or more concealment packets of audio data representative of audio data subsequent to said decoded audio data, and further storing said one or more generated concealment packets of audio data in said rendering buffer;
rendering said decoded audio data stored in said rendering buffer and said one or more generated concealment packets of audio data stored in said rendering buffer as audio output from said terminal device;
saving in said memory a corresponding state of the decoder prior to said generation of each of said concealment packets of audio data;
receiving a previous encoded audio signal packet using the packet receiver, said previous encoded audio signal packet being received after the receipt of the encoded audio signal packet, the previous encoded audio signal packet being comprised in the sequence of encoded audio signal packets prior to said encoded audio signal packet and being representative of audio data for which one of said concealment packets of audio data has been generated and stored in said rendering buffer;
restoring said saved state of said decoder corresponding to said generation of said concealment packet of audio data representative of said audio data represented by said previous encoded audio signal packet;
in response to receiving said previous encoded audio signal packet, using the decoder implemented on the processor to decode the received previous encoded audio signal packet to generate corresponding decoded audio data and storing said corresponding decoded audio data in the rendering buffer so as to replace said concealment packet of audio data representative of said audio data represented by said previous encoded audio signal packet which had been previously stored in said rendering buffer; and
retrieving said received encoded audio signal packet from said holding buffer and using the decoder implemented on the processor to re-decode the received encoded audio signal packet to generate corresponding decoded audio data and storing said corresponding decoded audio data in the rendering buffer;
wherein the memory further comprises a holding buffer, the computer readable medium product further comprising program instructions recorded thereupon for storing said received encoded audio signal packet in said holding buffer.
10. The computer readable medium product of claim 9 further comprising program instructions recorded thereupon for:
receiving a subsequent encoded audio signal packet using the packet receiver, said subsequent encoded audio signal packet being received after the receipt of the encoded audio signal packet, the subsequent encoded audio signal packet being comprised in the sequence of encoded audio signal packets subsequent to said encoded audio signal packet and being representative of audio data for which one of said concealment packets of audio data has been generated and stored in said rendering buffer;
restoring said saved state of said decoder corresponding to said generation of said concealment packet of audio data representative of said audio data represented by said subsequent encoded audio signal packet; and
in response to receiving said subsequent encoded audio signal packet, using the decoder implemented on the processor to decode the received subsequent encoded audio signal packet to generate corresponding decoded audio data and storing said corresponding decoded audio data in the rendering buffer so as to replace said concealment packet of audio data representative of said audio data represented by said subsequent encoded audio signal packet which had been previously stored in said rendering buffer.
11. The computer readable medium product of claim 10 further comprising program instructions recorded thereupon which, in further response to receiving said subsequent encoded audio signal packet, and without making any determination that one or more encoded audio signal packets in said sequence of encoded audio signal packets subsequent to said subsequent encoded audio signal packet have not been received, using the decoder to perform packet loss concealment to generate one or more concealment packets of audio data representative of audio data subsequent to said decoded audio data corresponding to said subsequent encoded audio signal packet, and further storing said one or more generated concealment packets of audio data in said rendering buffer.