Systems and methods for error resilience in video communication systems
Systems and methods for error resilient transmission and for random access in video communication systems are provided. The video communication systems are based on single-layer, scalable video, or simulcast video coding with temporal scalability, which may be used in video communication systems. A set of video frames or pictures in a video signal transmission is designated for reliable or guaranteed delivery to receivers using secure or high reliability links, or by retransmission techniques. The reliably-delivered video frames are used as reference pictures for resynchronization of receivers with the transmitted video signal after error incidence and for random access.
1. A system for media communications between a transmitting endpoint or server and one or more receiving endpoint(s) or server(s) over a packet-based communication network, the system comprising:
an encoder configured to use a threaded coding structure having a number of different layers including a lowest temporal layer, wherein each picture is associated with a picture index number;
wherein data corresponding to a single picture is portioned and transmitted in one or more individual data packets,
wherein an individual data packet comprises data elements that indicate:
for the lowest temporal layer pictures, a sequence number identifying said pictures,
for other temporal layer pictures, a reference to the sequence number of the most recent, in decoding order, lowest temporal layer picture, and
for all pictures, a ‘start’ flag and an ‘end’ flag that respectively indicate if the individual data packet contains the first or last data portions of the picture,
so that a receiving endpoint or server on receipt of the individual data packet can detect a loss of portions of a lowest temporal layer picture by examining if all data portions of the picture corresponding to the referenced picture index have been received at the receiving endpoint or server such that their packet sequence numbers contain no gaps, and where the first such data portion has the ‘start’ bit set and the last such data portion has the ‘end’ bit set.
2. The system of claim 1 wherein the data elements additionally indicate a series number associated with each spatial or quality layer, wherein the receiving endpoint or server detects if a lowest temporal layer picture of a particular spatial or quality layer is lost by determining if the picture corresponding to the referenced series number and sequence number has been received at the receiving endpoint or server.
3. The system of claim 1 comprising a codec conforming to H.264, wherein the lowest temporal layer pictures include pictures that are marked as long term reference pictures, and wherein decoding of at least a portion of the received media based on a lowest temporal layer picture by a decoder is controlled by Memory Management Control Operation (MMCO) commands.
4. The system of claim 1 wherein the communication network uses the Internet Protocol, media transport is performed using real-time protocol (RTP), and the data elements include data to indicate presence of a lowest temporal layer frame or fragment thereof in the packet.
5. The system of claim 4 wherein a receiving endpoint or server sends a negative acknowledgment message to the transmitting endpoint or server in response to the receiving endpoint's or server's detection of a lost lowest temporal layer picture or portion of such picture.
6. The system of claim 5 wherein the transmitting endpoint or server upon receiving the negative acknowledgment message retransmit the lost lowest temporal layer picture or portion of such picture.
7. The system of claim 1 , wherein the encoder conforms to H.264 Scalable Video Coding (SVC), and the data elements are carried in Network Adaptation Layer (NAL) unit header extension for SVC elements.
8. The system of claim 1 , wherein the encoder conforms to H.264 SVC, and the data elements are carried in Supplemental Enhancement Information (SEI) message.
9. A method for media communications between a transmitting endpoint or server and one or more receiving endpoint(s) or server(s) over a packet-based communication network, wherein an encoder is configured to use a threaded coding structure having a number of different layers including a lowest temporal layer, and wherein data corresponding to a single picture is portioned and transmitted in one or more individual data packets, the method comprising:
placing in each individual data packet data elements that indicate:
for the lowest temporal layer pictures, a sequence or index number identifying said pictures,
for other temporal layer pictures, a reference to the sequence number of the most recent, in decoding order, lowest temporal layer picture, and
for all pictures, a ‘start’ flag and an ‘end’ flag that indicate if the individual data packet contains the first or last, respectively, data portions of the picture, so that a receiving endpoint or server on receipt of the individual data packet can detect a loss of portions of a lowest temporal layer picture by examining if all data portions of the picture corresponding to the referenced picture index have been received at the receiving endpoint or server such that their packet sequence numbers contain no gaps, and where the first such data portion has the ‘start’ bit set and the last such data portion has the ‘end’ bit set.
10. The method of claim 9 wherein the data elements additionally indicate a series number associated with each spatial or quality layer, so that the receiving endpoint or server detects if a lowest temporal layer picture of a particular spatial or quality layer is lost by determining if the picture corresponding to the referenced series number and sequence number has been received at the receiving endpoint or server.
11. The method of claim 9 , wherein the transmitted media is coded using a codec conforming to H.264, wherein the lowest temporal layer pictures include pictures that are marked as long term reference pictures, and wherein decoding of at least a portion of the received media based on a lowest temporal layer picture by a decoder is controlled by Memory Management Control Operation (MMCO) commands.
12. The method of claim 9 wherein the communication network uses the Internet Protocol, media transport is performed using real-time transport protocol (RTP), and the data elements include data to indicate presence of a lowest temporal layer picture or fragment thereof in the packet.
13. The method of claim 12 wherein a receiving endpoint or server sends a negative acknowledgment message to the transmitting endpoint or server in response to the receiving endpoint's or server's detection of a lost lowest temporal layer picture or portion of such picture.
14. The method of claim 13 wherein the transmitting endpoint or server upon receiving the negative acknowledgment message retransmit the lost lowest temporal layer picture or portion of such picture.
15. The method of claim 14 , wherein the encoder conforms to H.264 Scalable Video Coding (SVC), and the data elements are carried in Network Adaptation Layer (NAL) unit header extension for SVC elements.
16. The method of claim 9 , wherein the encoder conforms to H.264 SVC, and the data elements are carried in Supplemental Enhancement Information (SEI) message.
17. A non-transitory computer readable medium comprising a set of instructions to direct a processor to perform the methods recited in one of claims 9 - 16 .