Reducing latency in video encoding and decoding
Techniques and tools for reducing latency in video encoding and decoding by constraining latency due to reordering of video frames, and by indicating the constraint on frame reordering latency with one or more syntax elements that accompany encoded data for the video frames. For example, a real-time communication tool with a video encoder sets a syntax element that indicates a constraint on frame reordering latency, which is consistent with inter-frame dependencies between multiple frames of a video sequence, then outputs the syntax element. A corresponding real-time communication tool with a video decoder receives the syntax element that indicates the constraint on frame reordering latency, determines the constraint on frame reordering latency based on the syntax element, and uses the constraint on frame reordering latency to determine when a reconstructed frame is ready for output (in terms of output order).
1. In a computer system that implements a video decoder, a method comprising:
receiving and parsing one or more syntax elements that indicate a constraint on frame reordering latency, wherein the constraint on frame reordering latency is expressed in terms of a maximum count of frames that can precede any frame of a video sequence in output order but follow that frame in coded order;
determining the constraint on frame reordering latency based upon the one or more syntax elements;
receiving encoded data for a given frame of the video sequence;
decoding the encoded data for the given frame to reconstruct the given frame; and
outputting the reconstructed frame, wherein the constraint is used to determine when the reconstructed frame is ready for output.
2. The method of claim 1 , wherein the one or more syntax elements and the encoded data for the given frame are signaled as part of syntax for a coded video bitstream, the method further comprising:
receiving and parsing a buffer size syntax element that indicates maximum size of a decoded picture buffer, wherein the buffer size syntax element is different than the one or more syntax elements that indicate the constraint on frame reordering latency.
3. The method of claim 1 , wherein the one or more syntax elements are signaled as part of a sequence parameter set.
4. The method of claim 1 , further comprising:
receiving a flag that indicates presence or absence of the one or more syntax elements.
5. The method of claim 1 , wherein one possible value of the one or more syntax elements indicates the constraint on frame reordering latency is undefined or has a default value, and wherein other possible values of the one or more syntax elements indicate an integer count for the constraint on frame reordering latency.
6. The method of claim 1 , wherein the video sequence includes multiple frames organized according to a temporal hierarchy, the multiple frames including the given frame, and wherein different constraints on frame reordering latencies apply for different temporal layers of the temporal hierarchy.
7. The method of claim 6 , further comprising selecting, as the constraint on frame reordering latency, one of the different constraints on frame reordering latencies depending on temporal resolution of the output.
8. The method of claim 1 , wherein a value of the one or more syntax elements indicates an integer count for the constraint on frame reordering latency relative to a maximum size of frame memory for reordering, the maximum size of frame memory for reordering being indicated with a different syntax element.
9. The method of claim 8 , wherein the constraint on frame reordering latency can be determined as a maximum count for the maximum size of frame memory for reordering plus the integer count for the constraint on frame reordering latency minus 1.
10. Computer-readable tangible storage having stored thereon computer-executable instructions for causing one or more processing units, when programmed thereby, to perform operations comprising:
setting one or more syntax elements that indicate a constraint on frame reordering latency, wherein the constraint on frame reordering latency is expressed in terms of a maximum count of frames that can precede any frame of a video sequence in output order but follow that frame in coded order;
outputting the one or more syntax elements, thereby facilitating determination of when reconstructed frames are ready for output;
encoding a given frame of the video sequence, thereby producing encoded data for the given frame, wherein the encoding uses inter-frame dependencies, within the video sequence, that are consistent with the constraint on frame reordering latency; and
outputting the encoded data for the given frame.
11. The computer-readable tangible storage of claim 10 , wherein the one or more syntax elements and the encoded data for the given frame are output as part of syntax for a coded video bitstream, the operations further comprising:
outputting a buffer size syntax element that indicates maximum size of a decoded picture buffer, wherein the buffer size syntax element is different than the one or more syntax elements that indicate the constraint on frame reordering latency.
12. The computer-readable tangible storage of claim 10 , wherein the one or more syntax elements are output as part of a sequence parameter set.
13. The computer-readable tangible storage of claim 10 , wherein one possible value of the one or more syntax elements indicates the constraint on frame reordering latency is undefined or has a default value, and wherein other possible values of the one or more syntax elements indicate an integer count for the constraint on frame reordering latency.
14. The computer-readable tangible storage of claim 10 , wherein a value of the one or more syntax elements indicate an integer count for the constraint on frame reordering latency relative to a maximum size of frame memory for reordering, the maximum size of frame memory for reordering being indicated with a different syntax element.
15. The computer-readable tangible storage of claim 14 , wherein the constraint on frame reordering latency can be determined as a maximum count for the maximum size of frame memory for reordering plus the integer count for the constraint on frame reordering latency minus 1.
16. One or more computer-readable media having stored thereon, in a bitstream, one or more syntax elements that indicate a constraint on frame reordering latency, wherein the constraint on frame reordering latency is expressed in terms of a maximum count of frames that can precede any frame of a video sequence in output order but follow that frame in coded order, the one or more computer-readable media further having stored thereon, in the bitstream, encoded data for a given frame of the video sequence, the one or more computer-readable media being selected from the group consisting of volatile memory, non-volatile memory, and storage devices, the bitstream being organized to facilitate playback, in a computer system that implements a video decoder, by operations comprising:
parsing the one or more syntax elements that indicate the constraint on frame reordering latency;
determining the constraint on frame reordering latency based upon the one or more syntax elements;
decoding the encoded data for the given frame to reconstruct the given frame; and
outputting the reconstructed frame, wherein the constraint is used to determine when the reconstructed frame is ready for output.
17. The one or more computer-readable media of claim 16 , wherein the one or more syntax elements are signaled as part of a sequence parameter set.
18. The one or more computer-readable media of claim 16 , wherein one possible value of the one or more syntax elements indicates the constraint on frame reordering latency is undefined or has a default value, and wherein other possible values of the one or more syntax elements indicate an integer count for the constraint on frame reordering latency.
19. The one or more computer-readable media of claim 16 , wherein a value of the one or more syntax elements indicates an integer count for the constraint on frame reordering latency relative to a maximum size of frame memory for reordering, the maximum size of frame memory for reordering being indicated with a different syntax element.
20. The one or more computer-readable media of claim 19 , wherein the constraint on frame reordering latency can be determined as a maximum count for the maximum size of frame memory for reordering plus the integer count for the constraint on frame reordering latency minus 1.