Supplemental enhancement information including confidence level and mixed content information
This application relates to video encoding and decoding, and specifically to tools and techniques for using and providing supplemental enhancement information in bitstreams. Among other things, the detailed description presents innovations for bitstreams having supplemental enhancement information (SEI). In particular embodiments, the SEI message includes picture source data (e.g., data indicating whether the associated picture is a progressive scan picture or an interlaced scan picture and/or data indicating whether the associated picture is a duplicate picture). The SEI message can also express a confidence level of the encoder's relative confidence in the accuracy of this picture source data. A decoder can use the confidence level indication to determine whether the decoder should separately identify the picture as progressive or interlaced and/or a duplicate picture or honor the picture source scanning information in the SEI as it is.
1. A computer system comprising:
a buffer, implemented using memory, configured to receive encoded data in a bitstream for pictures of a video sequence, the encoded data including a first flag and a second flag that is a separate syntax element from the first flag, the first flag indicating whether source scan type of the pictures of the video sequence is interlaced, and the second flag indicating whether the source scan type of the pictures of the video sequence is progressive, wherein the first flag and the second flag collectively and exclusively indicate one of the following unique states for the source scan type of the pictures of the video sequence: a state indicating that the source scan type of the pictures of the video sequence is interlaced, a state indicating that the source scan type of the pictures of the video sequence is progressive, a state indicating that the source scan type of the pictures of the video sequence is unknown, and a state indicating that the source scan type is independently indicated for each picture of the pictures of the video sequence by a value of a picture-level syntax element that is to be signaled as part of an SEI message or to be inferred; and
a video decoder, implemented using one or more processors, configured to perform operations to decode the encoded data to reconstruct at least some of the pictures of the video sequence, the operations including determining, for a given picture among the pictures of the video sequence, the value of a given picture-level syntax element that indicates the source scan type of the given picture, the value of the given picture-level syntax element indicating one of the following states: a state indicating that the source scan type of the given picture is interlaced, a state indicating that the source scan type of the given picture is progressive, and a state indicating that the source scan type of the given picture is unknown, wherein the determining the value of the given picture-level syntax element includes:
determining whether or not the given picture-level syntax element is present in the encoded data;
if the given picture-level syntax element is present in the encoded data, setting the value of the given picture-level syntax element according to the given picture-level syntax element in the encoded data; and
otherwise, the given picture-level syntax element not being present in the encoded data, inferring the value of the given picture-level syntax element, including:
if the first flag and the second flag indicate that the source scan type of the pictures of the video sequence is progressive, inferring the value of the given picture-level syntax element to indicate that the source scan type of the given picture is progressive;
if the first flag and the second flag indicate that the source scan type of the pictures of the video sequence is interlaced, inferring the value of the given picture-level syntax element to indicate that the source scan type of the given picture is interlaced; and
otherwise, inferring the value of the given picture-level syntax element to indicate that the source scan type of the given picture is unknown.
2. The computer system of claim 1 , wherein the encoded data in the bitstream further comprises a duplicate picture flag indicating whether one or more of the pictures are duplicate pictures.
3. The computer system of claim 1 , wherein the first flag and the second flag are part of information for a profile, level, and tier in the bitstream.
4. The computer system of claim 1 , wherein the SEI message is a picture-timing SEI message.
5. The computer system of claim 1 , wherein:
if the first flag is 0 and the second flag is 0, the source scan type of the pictures of the video sequence is unknown;
if the first flag is 0 and the second flag is 1, the source scan type of the pictures of the video sequence is progressive;
if the first flag is 1 and the second flag is 0, the source scan type of the pictures of the video sequence is interlaced; and
if the first flag is 1 and the second flag is 1, the source scan type is independently indicated for each picture of the pictures of the video sequence by the value of the picture-level syntax element for that picture.
6. The computer system of claim 1 , wherein the operations further comprise, as part of the inferring the value of the given picture-level syntax element:
if the first flag is 0 and the second flag is 0, the source scan type of the given picture is set to be unknown;
if the first flag is 0 and the second flag is 1, the source scan type of the given picture is set to be progressive; and
if the first flag is 1 and the second flag is 0, the source scan type of the given picture is set to be interlaced.
7. A computer system comprising:
a buffer configured to receive pictures of a video sequence; and
a video encoder configured to perform operations to encode the pictures of the video sequence, thereby producing encoded data in a bitstream, wherein the operations include:
determining a first flag and a second flag that is a separate syntax element from the first flag, the first flag indicating whether source scan type of the pictures of the video sequence is interlaced, and the second flag indicating whether the source scan type of the pictures of the video sequence is progressive, wherein the first flag and the second flag collectively and exclusively indicate one of the following unique states for the source scan type of the pictures of the video sequence: a state indicating that the source scan type of the pictures of the video sequence is interlaced, a state indicating that the source scan type of the pictures of the video sequence is progressive, a state indicating that the source scan type of the pictures of the video sequence is unknown, and a state indicating that the source scan type is independently indicated for each picture of the pictures of the video sequence by a value of a picture-level syntax element that is to be signaled as part of an SEI message or to be inferred;
outputting the first flag and the second flag as part of the encoded data in the bitstream;
determining, for a given picture among the pictures of the video sequence, the value of a given picture-level syntax element that indicates the source scan type of the given picture, the value of the given picture-level syntax element indicating one of the following states: a state indicating that the source scan type of the given picture is interlaced, a state indicating that the source scan type of the given picture is progressive, and a state indicating that the source scan type of the given picture is unknown, wherein, if the given picture-level syntax element is not present in the encoded data in the bitstream, the value of the given picture-level syntax element is to be inferred according to rules that include:
if the first flag and the second flag indicate that the source scan type of the pictures of the video sequence is progressive, inferring the value of the given picture-level syntax element to indicate that the source scan type of the given picture is progressive;
if the first flag and the second flag indicate that the source scan type of the pictures of the video sequence is interlaced, inferring the value of the given picture-level syntax element to indicate that the source scan type of the given picture is interlaced; and
otherwise, inferring the value of the given picture-level syntax element to indicate that the source scan type of the given picture is unknown; and
if the given picture-level syntax element is to be signaled, outputting the given picture-level syntax element as part of the encoded data in the bitstream.
8. The computer system of claim 7 , wherein the encoded data in the bitstream further comprises a duplicate picture flag indicating whether one or more of the pictures are duplicate pictures.
9. The computer system of claim 7 , wherein the first flag and the second flag are part of information for a profile, level, and tier in the bitstream.
10. The computer system of claim 7 , wherein the SEI message is a picture-timing SEI message.
11. The computer system of claim 7 , wherein:
if the first flag is 0 and the second flag is 0, the source scan type of the pictures of the video sequence is unknown;
if the first flag is 0 and the second flag is 1, the source scan type of the pictures of the video sequence is progressive;
if the first flag is 1 and the second flag is 0, the source scan type of the pictures of the video sequence is interlaced; and
if the first flag is 1 and the second flag is 1, the source scan type is independently indicated for each picture of the pictures of the video sequence by the value of the picture-level syntax element for that picture.
12. The computer system of claim 7 , wherein the given picture-level syntax element is not present in the encoded data, and wherein:
if the first flag is 0 and the second flag is 0, the source scan type of the given picture is set to be unknown;
if the first flag is 0 and the second flag is 1, the source scan type of the given picture is set to be progressive; and
if the first flag is 1 and the second flag is 0, the source scan type of the given picture is set to be interlaced.
13. The computer system of claim 7 , wherein the operations further comprise:
determining the source scan type of the pictures of the video sequence, including analyzing source scan characteristics of the pictures of the video sequence.
14. The computer system of claim 7 , wherein the operations further comprise:
determining the source scan type of the given picture, including analyzing source scan characteristics of the given picture.
15. One or more memory or storage devices having stored thereon encoded data in a bitstream for pictures of a video sequence, the encoded data having been produced, with a computer-implemented video encoder, by operations comprising:
determining a first flag and a second flag that is a separate syntax element from the first flag, the first flag indicating whether source scan type of the pictures of the video sequence is interlaced, and the second flag indicating whether the source scan type of the pictures of the video sequence is progressive, wherein the first flag and the second flag collectively and exclusively indicate one of the following unique states for the source scan type of the pictures of the video sequence: a state indicating that the source scan type of the pictures of the video sequence is interlaced, a state indicating that the source scan type of the pictures of the video sequence is progressive, a state indicating that the source scan type of the pictures of the video sequence is unknown, and a state indicating that the source scan type is independently indicated for each picture of the pictures of the video sequence by a value of a picture-level syntax element that is to be signaled as part of an SEI message or to be inferred;
outputting the first flag and the second flag as part of the encoded data in the bitstream;
determining, for a given picture among the pictures of the video sequence, the value of a given picture-level syntax element that indicates the source scan type of the given picture, the value of the given picture-level syntax element indicating one of the following states: a state indicating that the source scan type of the given picture is interlaced, a state indicating that the source scan type of the given picture is progressive, and a state indicating that the source scan type of the given picture is unknown, wherein, if the given picture-level syntax element is not present in the encoded data in the bitstream, the value of the given picture-level syntax element is to be inferred according to rules that include:
if the first flag and the second flag indicate that the source scan type of the pictures of the video sequence is progressive, inferring the value of the given picture-level syntax element to indicate that the source scan type of the given picture is progressive;
if the first flag and the second flag indicate that the source scan type of the pictures of the video sequence is interlaced, inferring the value of the given picture-level syntax element to indicate that the source scan type of the given picture is interlaced; and
otherwise, inferring the value of the given picture-level syntax element to indicate that the source scan type of the given picture is unknown; and
if the given picture-level syntax element is to be signaled, outputting the given picture-level syntax element as part of the encoded data in the bitstream.
16. The one or more memory or storage devices of claim 15 , wherein the encoded data in the bitstream further comprises a duplicate picture flag indicating whether one or more of the pictures are duplicate pictures.
17. The one or more memory or storage devices of claim 15 , wherein the first flag and the second flag are part of information for a profile, level, and tier in the bitstream.
18. The one or more memory or storage devices of claim 15 , wherein the SEI message is a picture-timing SEI message.
19. The one or more memory or storage devices of claim 15 , wherein:
if the first flag is 0 and the second flag is 0, the source scan type of the pictures of the video sequence is unknown;
if the first flag is 0 and the second flag is 1, the source scan type of the pictures of the video sequence is progressive;
if the first flag is 1 and the second flag is 0, the source scan type of the pictures of the video sequence is interlaced; and
if the first flag is 1 and the second flag is 1, the source scan type is independently indicated for each picture of the pictures of the video sequence by the value of the picture-level syntax element for that picture.
20. The one or more memory or storage devices of claim 15 , wherein the given picture-level syntax element is not present in the encoded data, and wherein:
if the first flag is 0 and the second flag is 0, the source scan type of the given picture is set to be unknown;
if the first flag is 0 and the second flag is 1, the source scan type of the given picture is set to be progressive; and
if the first flag is 1 and the second flag is 0, the source scan type of the given picture is set to be interlaced.