Optimized fast video frame repair for extreme low latency RTP delivery
Systems and methods are disclosed for modifying the encoding of a video in an ultra low-latency environment in response to a dropped or corrupted packet. A video source encodes the video using slice or tile encoding. Slice encoding divides each frame of the video into slices which are independently encoded. Each frame in a slice stream is either intra-coded or predictively encoded. Each slice or tile in an Intra frame, Predicted frame or bi-directional frame may be independently encoded as an Intra slice or tile, a predicted slice or tile or a Bi-directional slice or tile. Slices or tiles of each frame are multiplexed into data packets for transmission. In response to feedback indicating that a data packet was corrupted, is late or dropped, the video source determines what data was in the transmission data packet and modifies encoding of a subsequent slice or tile in the slice or tile stream to be intra-coded.
1 . A method comprising:
encoding, at an encoder, a video comprising a plurality of frames using slice encoding wherein the slice encoding comprises:
defining a slice structure comprising a plurality of slices, wherein each slice of the plurality of slices comprises pixel data for a respective same location in each frame of the plurality of frames; and
for each slice of the plurality of slices, separately encoding a respective slice stream, the encoded slice stream comprising, for each frame of the plurality of frames, one of: (a) an I-slice or (b) a P-slice, wherein:
each I-slice comprises all data for decoding the respective I-slice, and each P-slice comprises respective predictive data that can only be decoded using at least an I-slice or P-slice that precedes the respective P-slice in the respective slice stream;
transmitting a plurality of data packets to a client device, wherein each data packet of the plurality of data packets comprises encoded slice data of at least one slice of a frame of the plurality of frames;
establishing a direct connection between the encoder and a decoder executing on the client device, wherein the direct connection is established as a separate connection in addition to a connection for transmitting the video from the encoder to the client device;
receiving feedback data from the client device indicating that a particular slice in the particular frame was not received or was corrupted by receiving an Application Programming Interface (API) communication from the decoder via the direct connection; and
in response to the receiving the feedback data:
modifying the encoding of the video such that a slice in a next frame to be encoded in the slice stream of the particular slice is an I-slice.
2 . The method of claim 1 , wherein the plurality of data packets comprises a plurality of video data packets, and wherein the receiving the feedback data from the client device indicating that the particular slice in the particular frame was not received or was corrupted comprises:
receiving an indication that a particular video data packet of the plurality of data packets was not received in time at the client device, wherein the particular video data packet comprises the particular slice in the particular frame.
3 . The method of claim 2 , further comprising:
in response to the receiving the feedback data:
refraining from retransmitting the particular video data packet.
4 . A method:
encoding, at an encoder, a video comprising a plurality of frames using slice encoding wherein the slice encoding comprises:
defining a slice structure comprising a plurality of slices, wherein each slice of the plurality of slices comprises pixel data for a respective same location in each frame of the plurality of frames; and
for each slice of the plurality of slices, separately encoding a respective slice stream, the encoded slice stream comprising, for each frame of the plurality of frames, one of: (a) an I-slice or (b) a P-slice, wherein:
each I-slice comprises all data for decoding the respective I-slice, and each P-slice comprises respective predictive data that can only be decoded using at least an I-slice or P-slice that precedes the respective P-slice in the respective slice stream;
transmitting a plurality of data packets to a client device, wherein each data packet of the plurality of data packets comprises encoded slice data of at least one slice of a frame of the plurality of frames;
receiving feedback data from the client device indicating that (a) a particular slice in the particular frame was not received or was corrupted and (b) an audio data packet was not received or was corrupted; and
in response to receiving the feedback data:
modifying the encoding of the video such that a slice in a next frame to be encoded in the slice stream of the particular slice is an I-slice; and
retransmitting the audio data packet.
5 . The method of claim 1 , wherein the defining the slice structure comprises:
defining a plurality of tiles, wherein a tile comprises pixel data for a rectangular region of each frame of the plurality of frames.
6 . The method of claim 1 , wherein the defining the slice structure comprises:
defining each slice of the plurality of slices as a sub-partition wherein a sub-partition comprises pixel data for a region of each frame of the plurality of frames.
7 . The method of claim 1 , wherein the plurality of frames comprises a plurality of sequential frames, wherein each frame is associated with a sequence number, wherein the plurality of data packets comprises a plurality of video data packets and wherein the method further comprises:
assembling a packetized elementary stream (PES) packet for each frame of the plurality of frames, wherein the PES packet comprises all slices of a frame of the plurality of sequential frames with a particular sequence number.
8 . The method of claim 7 , further comprising:
assembling a plurality of Real-Time Transport Protocol (RTP) data packets by multiplexing the PES packet into the plurality of RTP data packets, wherein each RTP data packet of the plurality of RTP data packets comprises a subset of all the slices of the frame with the particular sequence number and wherein each RTP data packet is associated with an RTP sequence number;
storing information indicating the subset of all the slices of the frame with the particular sequence number stored in each RTP data packet.
9 . The method of claim 7 , further comprising:
storing a data structure that defines for each RTP data packet of the plurality of RTP data packets:
(a) a respective RTP sequence number;
(b) identification of slices multiplexed into the respective RTP data packet; and
(c) a sequence number of a frame of the slices identified by the identification of slices.
10 . The method of claim 9 , wherein the receiving the feedback data from the client device indicating that the particular slice in the particular frame was not received or was corrupted comprises:
receiving from the client device a particular RTP sequence number; and
cross referencing the particular RTP sequence number with the data structure to identify the particular slice.
11 . The method of claim 9 , wherein the identification of slices multiplexed into the respective RTP data packet comprises:
selecting each slice of the slices multiplexed into the respective RTP data packet such that each slice comprises pixel data for a plurality of pixels located directly adjacent to another slice in the slices multiplexed into the respective RTP data packet.
12 . The method of claim 9 , wherein the identification of slices multiplexed into the respective RTP data packet comprises:
selecting each slice of the slices multiplexed into the respective RTP data packet such that each slice comprises pixel data for a plurality of pixels and is selected randomly.
13 . The method of claim 1 , further comprising, in response to the receiving the feedback data:
determining a number of slices in the particular frame which were not received or corrupted;
in response to determining that the number of slices in the particular frame which were not received or corrupted is larger than a threshold value, modifying the encoding of the video such that a next frame is encoded as an I-frame, wherein each slice in the particular frame are encoded as an I-slice.
14 . A system comprising:
an encoder configured to:
encode a video comprising a plurality of frames using slice encoding wherein the slice encoding comprises:
defining a slice structure comprising a plurality of slices, wherein each slice of the plurality of slices comprises pixel data for a respective same location in each frame of the plurality of frames; and
for each slice of the plurality of slices, separately encoding a respective slice stream, the encoded slice stream comprising, for each frame of the plurality of frames, one of: (a) an I-slice or (b) a P-slice, wherein:
each I-slice comprises all data for decoding the respective I-slice, and each P-slice comprises respective predictive data that can only be decoded using at least an I-slice or P-slice that precedes the respective P-slice in the respective slice stream;
input/output circuitry configured to:
transmit a plurality of data packets to a client device, wherein each data packet of the plurality of data packets comprises encoded slice data of at least one slice of a frame of the plurality of frames;
receive feedback data from the client device indicating that a particular slice in the particular frame was not received or was corrupted; and
control circuitry configured to:
in response to the receiving the feedback data:
modify the encoding of the video such that a slice in a next frame to be encoded in the slice stream of the particular slice is an I-slice.
15 . The system of claim 14 , wherein the plurality of data packets comprises a plurality of video data packets, and wherein the input/output circuitry receives the feedback data from the client device indicating that the particular slice in the particular frame was not received or was corrupted by:
receiving an indication that a particular video data packet of the plurality of data packets was not received in time at the client device, wherein the particular video data packet comprises the particular slice in the particular frame.
16 . The system of claim 15 , wherein the control circuitry is further configured to, in response to the receiving the feedback data:
refrain from retransmitting the particular video data packet.
17 . The system of claim 14 , wherein the input/output circuitry is further configured to:
receive feedback data from the client device indicating that an audio data packet was not received or was corrupted; and
in response to receiving the feedback data:
retransmit the audio data packet.
18 . The system of claim 14 , wherein the encoder is further configured to define the slice structure by:
defining a plurality of tiles, wherein a tile comprises pixel data for a rectangular region of each frame of the plurality of frames.