Skipped video data recovery using multiple alternative recovery modes
View Patent ↗In some embodiments, a video decoder is capable of recovering skipped video data (e.g. motion vectors, prediction modes, quantization parameters, selected frequency-domain coefficients such as DC and 5-lowest-frequency AC coefficients, and/or entire video data blocks) using multiple alternative recovery modes such as spatial interpolation, temporal interpolation, and motion search. To decide whether to skip a particular data type for a block, the encoder evaluates the effect of skipping the data on rate and distortion by simulating the decoder data recovery using the multiple recovery modes. The encoder transmits indicators of skipped data types and associated recovery modes, if different from decoder baselines. The skipped data and recovery mode indicators may be included in macroblock and/or slice headers, and/or as part of sequence, group-of-picture (GOP), or picture coding parameter data. Skipped data recovery decoder resources (e.g. motion search logic) may be used for recovering data lost due to transmission errors.
1. A video encoding method comprising:
receiving a sequence of video frames, each frame comprising a plurality of non-overlapping rectangular blocks;
encoding the sequence of video frames to generate an encoded video data set for each of the plurality of non-overlapping rectangular blocks included in the video frames, wherein the encoded video data set comprises a block encoding mode, a motion vector, a quantization parameter, and a set of frequency domain coefficients for each image block;
determining, on a block-by-block basis, a decoding consequence of skipping transmission to a decoder of a target subset of the encoded data set for each image block by analyzing a predicted recovery of the target subset by the decoder according to a plurality of alternative decoder recovery modes, wherein the target subset includes at least one block-level parameter set in the encoded data set; and
deciding, on a block-by-block basis, whether to skip transmission of the target subset to the decoder according to the decoding consequence of skipping transmission of the target subset to the decoder.
2. The method of claim 1 , wherein the plurality of alternative recovery modes includes an interpolation mode.
3. The method of claim 2 , wherein the interpolation mode is a spatial interpolation mode.
4. The method of claim 2 , wherein the interpolation mode is a temporal interpolation mode.
5. The method of claim 1 , wherein the plurality of alternative recovery modes includes at least one of a spatial interpolation mode and a temporal interpolation mode.
6. The method of claim 1 , wherein the target subset includes at least one of the block encoding mode, the motion vector, the quantization parameter, and a partial subset of the set of frequency domain coefficients.
7. The method of claim 6 , wherein the partial subset of the set of frequency domain coefficients consists of a DC coefficient and a set of 5-lowest-frequency AC coefficients in zig-zag order.
8. The method of claim 1 , further comprising transmitting to the decoder an indicator that transmission of the target subset has been skipped, as it is decided to skip transmission of the target subset to the decoder.
9. The method of claim 1 , further comprising transmitting to the decoder an indicator of a selected recovery mode to be used by the decoder to recover the target subset.
10. The method of claim 1 , further comprising assembling a video data packet including a header and a video data payload, wherein the header includes an indicator that transmission of the target subset has been skipped.
11. The method of claim 1 , wherein determining the decoding consequence of skipping transmission of the target subset to the decoder comprises determining a distortion resulting from skipping transmission of the target subset to the decoder.
12. The method of claim 1 , comprising deciding whether to skip transmission of the target subset to the decoder of according to an indicator of a loss rate for an encoder-decoder communications channel.
13. The method of claim 1 , further comprising:
determining a decoding consequence of entirely skipping transmission of the video image block to the decoder by analyzing a predicted recovery of the video image block by the decoder according to the plurality of alternative decoder recovery modes; and
deciding whether to entirely skip transmission of the video image block to the decoder according to the decoding consequence of entirely skipping transmission of the video image block to the decoder.
14. The method of claim 6 , wherein the target subset to be skipped varies on a block-by-block basis within each frame.
15. A video data encoder comprising:
an input buffer to receive a sequence of video frames, each frame comprising a plurality of non-overlapping rectangular blocks;
video encoding logic configured to encode the sequence of video frames to generate an encoded video data set for each of the plurality of non-overlapping blocks in the video frames, wherein the encoded video data set comprises a block encoding mode, a motion vector, a quantization parameter, and a set of frequency domain coefficients for each image block; and
at least one data reduction unit connected to the video encoding logic and configured to determine, on a block-by-block basis, a decoding consequence of skipping transmission to a decoder of a target subset of the encoded data set for each image block by analyzing a predicted recovery of the target subset by the decoder according to a plurality of alternative decoder recovery modes, wherein the target subset includes at least one block-level parameter set in the encoded data set, said at least one data reduction unit further configured to decide, on a block-by-block basis, whether to skip transmission of the target subset to the decoder according to the decoding consequence of skipping transmission of the target subset to the decoder.
16. A video encoding apparatus comprising:
means for receiving a sequence of video frames, each frame comprising a plurality of non-overlapping rectangular blocks;
means for encoding the sequence of video frames to generate an encoded video data set for each of the plurality of non-overlapping blocks included in one video frame, wherein the encoded video data set comprises a block encoding mode, a motion vector, a quantization parameter, and a set of frequency domain coefficients for each block;
means for determining, on a block-by-block basis, a decoding consequence of skipping transmission to a decoder of a target subset of the encoded data set for each image block by analyzing a predicted recovery of the target subset by the decoder according to a plurality of alternative decoder recovery modes, wherein the target subset includes at least one block-level parameter set in the encoded data set; and
means for deciding, on a block-by-block basis, whether to skip transmission of the target subset to the decoder according to the decoding consequence of skipping transmission of the target subset to the decoder.
17. A video encoding method comprising:
encoding a plurality of non-overlapping rectangular blocks in a video frame to generate an encoded video data set for each image block;
prioritizing data for transmission in subsets of regional blocks included in the frame, based on effects of region loss on system distortion; and
determining, on a block-by-block basis, a decoding consequence of skipping transmission to a decoder of a target subset of the encoded data set for each image block by analyzing a predicted recovery of the target subset by the decoder according to a plurality of alternative decoder recovery modes.
18. A video decoding method comprising:
receiving from a video encoder an encoded video data set for each image block of a plurality of non-overlapping rectangular blocks in a video frame, wherein the video data set comprises a block encoding mode, a motion vector, a quantization parameter, a set of frequency domain coefficients, and an indicator of skipped data;
selecting a recovery mode of a plurality of alternative recovery modes, wherein a recovery mode is selected on a block-by-block basis;
employing the video data set for each image block to recover a recovery data set for a respective video image block according to the recovery mode, the recovery data set including at least one block-level parameter set associated with the video data set; and
generating a decoded sequence of video frames from the recovery data set and the video data set.
19. The method of claim 18 , wherein the plurality of alternative recovery modes includes an interpolation mode.
20. The method of claim 19 , wherein the interpolation mode is a spatial interpolation mode.
21. The method of claim 19 , wherein the interpolation mode is a temporal interpolation mode.
22. The method of claim 18 , wherein the plurality of alternative recovery modes includes at least one of a spatial interpolation mode and a temporal interpolation mode.
23. The method of claim 18 , wherein the recovery data set includes the block encoding mode.
24. The method of claim 18 , wherein the recovery data set includes the motion vector.
25. The method of claim 18 , wherein the recovery data set includes the quantization parameter.
26. The method of claim 18 , wherein the recovery data set includes a partial subset of the set of frequency domain coefficients.
27. The method of claim 26 , wherein the partial subset of the set of frequency domain coefficients consists of a DC coefficient and a set of 5-lowest-frequency AC coefficients in zig-zag order.
28. The method of claim 18 , further comprising receiving from the encoder an identifier of the recovery data set to be recovered.
29. The method of claim 18 , further comprising receiving from the encoder an indicator of a selected recovery mode to be used by the decoder to recover the recovery data set.
30. The method of claim 18 , further comprising receiving from the encoder a video data packet including a header and a video data payload, wherein the header includes an identifier of the recovery data set to be recovered.
31. The method of claim 18 , wherein the recovery data set includes an entire video image block.
32. A video decoder comprising:
an input buffer to receive encoded data as a sequence of input video data packets from an encoder;
at least one data recovery unit configured to select a recovery mode from a plurality of recovery modes, employ a video data set encoding a the video sequence to recover a recovery data set according to the recovery mode, a recovery mode being selected on a block-by-block basis, the recovery data set including at least one block-level parameter set associated with the video data set; and
decoding logic connected to the data recovery unit and configured to generate a decoded sequence of video frames from the recovery data set and the video data set;
wherein a frame in the video sequence comprises a plurality of non-overlapping rectangular blocks including a video image block, and wherein the video data set comprises a block encoding mode, a motion vector, a quantization parameter, a set of frequency domain coefficients associated with the video image block, and an indicator of skipped data.
33. A video decoding apparatus comprising:
means for receiving from a video encoder a video data set for each encoded block of a plurality of non-overlapping rectangular blocks included in a video frame, wherein each video data set comprises a block encoding mode, a motion vector, a quantization parameter, a set of frequency domain coefficients, and an indicator of skipped data;
means for selecting a recovery mode of a plurality of alternative recovery modes, a recovery mode being selected on a block-by-block basis;
means for employing the video data set to recover a recovery data set for a video image block according to the recovery mode, the recovery data set including at least one block-level parameter set associated with the video data set; and
means for generating a decoded sequence of video frames from the recovery data set and the video data set.