IP Library › Granted Patent US 12,389,042
Granted Patent B2
US 12,389,042 · App. 17/464,364 · Granted Aug 12, 2025

Pictures and layers included in a VVC image item

Inventors: Ye-Kui Wang (San Diego, CA); Li Zhang (San Diego, CA)
H04N19/70H04N19/107H04N19/159H04N19/176H04N19/184H04N19/517H04N19/67
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,389,042
App. No.
17/464,364
Granted
Aug 12, 2025
Kind
B2
Abstract

Systems, methods and apparatus for processing image data are described. One example method includes performing a conversion between a visual media file and a bitstream. The visual media file comprises a sequence of one or more pictures according to a media file format, and the bitstream comprises one or more access units according to a video coding format. The bitstream is coded according to the video coding format. The media file format specifies that an image item of a specific type value in the visual media file includes a single access unit of the bitstream. The single access unit is either an Intra Random Access Picture (IRAP) access unit or a Gradual Decoding Refresh (GDR) access unit according to the video coding format. All pictures in the GDR access unit are identified as a recovery point in the bitstream.

Claims (41)

1. A method for processing image data, comprising:

performing a conversion between a visual media file and a bitstream, wherein the visual media file comprises a sequence of one or more pictures according to a media file format, and wherein the bitstream comprises one or more access units according to a video coding format,

wherein the bitstream is coded according to the video coding format,

wherein the media file format specifies that an image item of a specific type value in the visual media file includes a single access unit, wherein the single access unit is allowed to be one of a single Intra Random Access Picture (IRAP) access unit according to the video coding format and a single Gradual Decoding Refresh (GDR) access unit according to the video coding format, and wherein all pictures in the GDR access unit are identified as a recovery point in the bitstream;

wherein the media file format specifies that an image item of a specific type value in the visual media file excludes layers that do not belong to a target output layer set; and

wherein the media file format further specifies that, in response to a record of operating points being included in a property descriptor of the operating points that indicates high-level characteristics of the bitstream, a value of a first syntax element ptl_max_temporal_id in the record and a value of a second syntax element max_temporal_id in the record are constrained to be equal to zero, wherein the first syntax element indicates a maximum temporal identification of layer representation for a specified profile, tier, and level structure, and wherein the second syntax element indicates a maximum temporal identification for an operating point.

2. The method of claim 1 , wherein the video coding format corresponds to a Versatile Video Coding standard according to ISO/IEC 23090-3, and wherein the specific type value is specified as ‘vvc1’.

3. The method of claim 1 , wherein each of the all pictures in the GDR access unit includes a picture header field having a value zero indicating that a corresponding picture is the recovery point.

4. The method of claim 3 , wherein the picture header field corresponds to a ph_recovery_poc_cnt field.

5. The method of claim 1 , wherein the image item includes layers in an output layer set identified by a property indicating the target output layer set and does not include other layers.

6. The method of claim 1 , wherein the media file format specifies that an image item of a specific type value in the visual media file includes at least part of an access unit in which a picture comprises one or more subpictures.

7. The method of claim 6 , wherein the image item includes an entirety of the access unit.

8. The method of claim 6 , wherein the image item includes part of the access unit, and wherein, for each layer present in the bitstream, the one or more subpictures form a rectangular region.

9. The method of claim 1 , wherein the conversion includes encoding the visual media file into the bitstream.

10. The method of claim 1 , wherein the conversion includes decoding the visual media file from the bitstream.

11. An apparatus for processing visual media file data comprising a processor and a non-transitory memory with instructions thereon, wherein the instructions upon execution by the processor, cause the processor to:

perform a conversion between a visual media file and a bitstream, wherein the visual media file comprises a sequence of one or more pictures according to a media file format, and wherein the bitstream comprises one or more access units according to a video coding format,

wherein the bitstream is coded according to the video coding format,

wherein the media file format specifies that an image item of a specific type value in the visual media file includes a single access unit, wherein the single access unit is allowed to be one of a single Intra Random Access Picture (IRAP) access unit according to the video coding format and a single Gradual Decoding Refresh (GDR) access unit according to the video coding format, and wherein all pictures in the GDR access unit are identified as a recovery point in the bitstream;

wherein the media file format specifies that an image item of a specific type value in the visual media file excludes layers that do not belong to a target output layer set; and

wherein the media file format further specifies that, in response to a record of operating points being included in a property descriptor of the operating points that indicates high-level characteristics of the bitstream, a value of a first syntax element ptl_max_temporal_id in the record and a value of a second syntax element max_temporal_id in the record are constrained to be equal to zero, wherein the first syntax element indicates a maximum temporal identification of layer representation for a specified profile, tier, and level structure, and wherein the second syntax element indicates a maximum temporal identification for an operating point.

12. The apparatus of claim 11 , wherein the video coding format corresponds to a Versatile Video Coding standard according to ISO/IEC 23090-3, and wherein the specific type value is specified as ‘vvc1’.

13. The apparatus of claim 11 , wherein each of the all pictures in the GDR access unit includes a picture header field having a value zero indicating that a corresponding picture is the recovery point, and wherein the picture header field corresponds to a ph_recovery_poc_cnt field.

14. The apparatus of claim 11 , wherein the media file format specifies that the image item of the specific type value in the visual media file includes layers in an output layer set identified by a property indicating the target output layer set and does not include other layers.

15. The apparatus of claim 11 , wherein the media file format specifies that an image item of a specific type value in the visual media file includes at least part of an access unit in which a picture comprises one or more subpictures.

16. The apparatus of claim 15 , wherein the image item includes an entirety of the access unit.

17. The apparatus of claim 15 , wherein the image item includes part of the access unit, and wherein, for each layer present in the bitstream, the one or more subpictures form a rectangular region.

18. A non-transitory computer-readable storage medium storing instructions that cause a processor to:

perform a conversion between a visual media file and a bitstream, wherein the visual media file comprises a sequence of one or more pictures according to a media file format, and wherein the bitstream comprises one or more access units according to a video coding format,

wherein the bitstream is coded according to the video coding format,

wherein the media file format specifies that an image item of a specific type value in the visual media file includes a single access unit, wherein the single access unit is allowed to be one of a single Intra Random Access Picture (IRAP) access unit according to the video coding format and a single Gradual Decoding Refresh (GDR) access unit according to the video coding format, and wherein all pictures in the GDR access unit are identified as a recovery point in the bitstream;

wherein the media file format specifies that an image item of a specific type value in the visual media file excludes layers that do not belong to a target output layer set; and

wherein the media file format further specifies that, in response to a record of operating points being included in a property descriptor of the operating points that indicates high-level characteristics of the bitstream, a value of a first syntax element ptl_max_temporal_id in the record and a value of a second syntax element max_temporal_id in the record are constrained to be equal to zero, wherein the first syntax element indicates a maximum temporal identification of layer representation for a specified profile, tier, and level structure, and wherein the second syntax element indicates a maximum temporal identification for an operating point.

19. The non-transitory computer-readable storage medium of claim 18 , wherein each of the all pictures in the GDR access unit includes a picture header field having a value zero indicating that a corresponding picture is the recovery point, and wherein the picture header field corresponds to a ph_recovery_poc_cnt field.

20. A non-transitory computer-readable recording medium storing a bitstream of a video which is generated by a method performed by a video processing apparatus, wherein the method comprises:

generating a visual media file from the bitstream of the video,

wherein the visual media file comprises a sequence of one or more pictures according to a media file format, and wherein the bitstream comprises one or more access units according to a video coding format,

wherein the bitstream is coded according to the video coding format,

wherein the media file format specifies that an image item of a specific type value in the visual media file includes a single access unit, wherein the single access unit is allowed to be one of a single Intra Random Access Picture (IRAP) access unit according to the video coding format and a single Gradual Decoding Refresh (GDR) access unit according to the video coding format, and wherein all pictures in the GDR access unit are identified as a recovery point in the bitstream;

wherein the media file format specifies that an image item of a specific type value in the visual media file excludes layers that do not belong to a target output layer set; and

wherein the media file format further specifies that, in response to a record of operating points being included in a property descriptor of the operating points that indicates high-level characteristics of the bitstream, a value of a first syntax element ptl_max_temporal_id in the record and a value of a second syntax element max_temporal_id in the record are constrained to be equal to zero, wherein the first syntax element indicates a maximum temporal identification of layer representation for a specified profile, tier, and level structure, and wherein the second syntax element indicates a maximum temporal identification for an operating point.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 1, 2021
From: WANG, YE-KUI; ZHANG, LI
To: BYTEDANCE INC.
Reel/Frame 057360/0202 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 1, 2021
From: BYTEDANCE INC.
To: LEMON INC.
Reel/Frame 057360/0222 →
Continuity (2)
Provisional Application 63073829 · Sep 2, 2020
Related Publication 20220070495A1 · Mar 3, 2022
References Cited (87)
US 8365060B2 · Wang et al. · 2013 [cited by applicant]
US 9357199B2 · Wang et al. · 2016 [cited by applicant]
US 9584792B2 · Wang et al. · 2017 [cited by applicant]
US 9648299B2 · Wang et al. · 2017 [cited by applicant]
US 10349067B2 · Hendry et al. · 2019 [cited by applicant]
US 10791315B2 · Wang et al. · 2020 [cited by applicant]
US 10873736B2 · Wang et al. · 2020 [cited by applicant]
US 11563983B2 · Wang et al. · 2023 [cited by applicant]
US 11595672B2 · Wang et al. · 2023 [cited by applicant]
US 11863768B2 · Wang et al. · 2024 [cited by applicant]
US 20050062888A1 · Wood et al. · 2005 [cited by applicant]
US 20060265643A1 · Saft et al. · 2006 [cited by applicant]
US 20080001945A1 · Kashito et al. · 2008 [cited by applicant]
US 20090085918A1 · Hollingworth et al. · 2009 [cited by applicant]
US 20150110167A1 · Chen et al. · 2015 [cited by applicant]
US 20150264373A1 · Wang et al. · 2015 [cited by applicant]
US 20160073119A1 · Toma et al. · 2016 [cited by applicant]
US 20160132221A1 · Lasser et al. · 2016 [cited by applicant]
US 20160373771A1 · Hendry et al. · 2016 [cited by applicant]
US 20170237999A1 · Hendry et al. · 2017 [cited by applicant]
US 20180103199A1 · Hendry et al. · 2018 [cited by applicant]
US 20180103271A1 · Wang et al. · 2018 [cited by applicant]
US 20200336726A1 · Wang et al. · 2020 [cited by applicant]
US 20210392361A1 · Hendry et al. · 2021 [cited by applicant]
US 20220070480A1 · Wang et al. · 2022 [cited by applicant]
US 20220070496A1 · Wang et al. · 2022 [cited by applicant]
US 20220070497A1 · Wang et al. · 2022 [cited by applicant]
US 20230027058A1 · Hannuksela et al. · 2023 [cited by applicant]
US 20230059074A1 · Drugeon · 2023 [cited by examiner]
CN 101313577A · 2008 [cited by applicant]
CN 101981931A · 2011 [cited by applicant]
CN 102428439A · 2012 [cited by applicant]
CN 104202661A · 2014 [cited by applicant]
CN 111182308A · 2020 [cited by applicant]
JP 2006079748A · 2006 [cited by applicant]
JP 2008530717A · 2008 [cited by applicant]
JP 2009512325A · 2009 [cited by applicant]
JP 2009182891A · 2009 [cited by applicant]
JP 2016540416A · 2016 [cited by applicant]
JP 2019506096A · 2019 [cited by applicant]
TW I687087B · 2020 [cited by applicant]
WO 2010058962A2 · 2010 [cited by applicant]
WO 2016057884A1 · 2016 [cited by applicant]
WO 2020130922A1 · 2020 [cited by applicant]
“Information technology—High efficiency coding and media delivery in heterogeneous environments—Part 12: Image File Format—amendment 3; Support for VVC, EVC, Slideshow and other Improvement” IOS/IEC 23008-12:2017(E) Ame… [cited by examiner]
Hannuksela et al. “VVC Subpictures in ISO/IEC 14496-15” 129 MPEG Meeting, Jan. 13-17, 2020, Brussels, Motion Picture Expert Group or ISO/IEC JTC 1/SC 29/WG 11, No. m52209, Jan. 8, 2020, XP030224811, retrieved from the i… [cited by applicant]
“Information Technology—High Efficiency Coding and Media Delivery in Heterogeneous Environments—Part 12: Image File Format,” ISO/IEC 23008-12, ISO/IEC Standard, Nov. 30, 2017, pp. 1-72, XP009522782, retreived from the i… [cited by applicant]
“Information Technology—High Efficiency Coding and Media Delivery in Heterogeneous Environments—Part 12: Image File Format—Amendment 3: Support for VVC, EVC, Slideshows and Other Improvements,” Text of ISO/IEC 23008-12:… [cited by applicant]
“Information Technology—Coding of Audio-Visual Objects—Part 15: Carriage Network Abstraction Later (NAL) Unit Structured Video in the ISO Base Media File Format,” 128 MPEG Meeting, Oct. 7-11, 2019, Geneva, Motion Pictur… [cited by applicant]
“Technologies Under Consideration on ISO/IEC 23008-12,” 128 MPEG Meeting, Oct. 7-11, 2019, Geneva, Motion Picture Expert Group or ISO8IEC JTC 1/SC 29/WG 11, No. n18819, Nov. 20, 2019, XP030225535, retrieved from the int… [cited by applicant]
Wang et al. “On Slideshow Transition Properties,” 132 MPEG Meeting, Oct. 12-16, 2020, Motion Picture Expert Group or ISO/IEC JTC 1/SC 29/WG 11, No. m55170, Oct. 5, 2020, XP030292687, retrieved from the internet Oct. 5, … [cited by applicant]
Wang et al. “Comments on Image File Formats for VVC, EVC, and Slideshow,” 132 MPEG Meeting Oct. 12-16, 2020, Motion Picture Expert Group or ISO/IEC JTC 1/SC 29/WG 11, No. m55174, Oct. 4, 2020, XP030292689, retrieved fro… [cited by applicant]
Extended European Search Report from European Patent Application No. 21194356.8 dated Jan. 28, 2022 (11 pages). [cited by applicant]
Extended European Search Report from European Patent Application No. 21194357.6 dated Jan. 28, 2022 (12 pages). [cited by applicant]
Extended European Search Report from European Patent Application No. 21194359.2 dated Jan. 28, 2022 (11 pages). [cited by applicant]
Extended European Search Report from European Patent Application No. 21194361.8 dated Feb. 2, 2022 (8 pages). [cited by applicant]
Boyce et al. ““Versatile Supplemental Enhancement Information Messages for Coded Video Bitstreams (Draft 5),”” Joint Video Experts Team (JVET)of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 1118th Meeting: by teleconfere… [cited by applicant]
Bross et al. ““Versatile Video Coding (Draft 10),”” Joint Video Experts Team (JVET)of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 1119th Meeting: by teleconference, Jun. 22-Jul. 1, 2020, document JVET-S2001, 2020. [cited by applicant]
Chen et al. “Algorithm Description of Joint Exploration Test Model 7 (JEM 7),” Joint Video Exploration Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 7th Meeting: Torino, IT, Jul. 13-21, 2017, document J… [cited by applicant]
“Information Technology—Coding of Audio-Visual Objects—Part 15: Carriage of Network Abstraction Layer (NAL) Unit Structured Video in the ISO Base Media File Format—Amendment 2: Carriage of VVC and EVC in ISOBMFF” ISO/IE… [cited by applicant]
“High Efficiency Video Coding,” Series H: Audiovisual and Multimedia Systems: Infrastructure of Audiovisual Services—Coding of Moving Video, ITU-T Telecommunication Standardization Sector of ITU, H.265, Feb. 2018. [cited by applicant]
“Information Technology—Coded Representation of Immersive Media—Part 3: Versatile Video Coding” ISO/IEC JTC 1/SC 29 /WG 11 N 18692, Text of ISO/IEC CD 23090-3, ISO/IEC JTC 1/SC 29/WG 11, Jul. 12, 2019. [cited by applicant]
“Information Technology—High Efficiency Coding and Media Delivery in Heterogeneous Environments—Part 12: Image File Format—Amendment 3: Support for VVC, EVC, Slideshows and Other Improvements,” ISO/IEC 23008-12:2017(E) … [cited by applicant]
“Versatile Supplemental Enhancement Information Messages for Coded Video Bitstreams,” Series H: Audiovisual and Multimedia Systems: Infrastructure of Audiovisual Services—Coding of Moving Video, ITU-T Telecommunication … [cited by applicant]
Examination Report from Indian Patent Application No. 202144039697 dated Mar. 24, 2022 (6 pages). [cited by applicant]
Non-Final Office Action from U.S. Appl. No. 17/464,251 dated May 13, 2022. [cited by applicant]
Non-Final Office Action from U.S. Appl. No. 17/464,415 dated Jun. 2, 2022. [cited by applicant]
Non-Final Office Action from U.S. Appl. No. 17/464,311 dated Jun. 15, 2022. [cited by applicant]
Bross et al. “Versatile Video Coding (Draft 10),” Joint Video Experts Team (JVET)of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 1119th Meeting: by teleconference, Jun. 22-Jul. 1, 2020, document JVET-S0269, 2020. (cited … [cited by applicant]
“Information Technology—High Efficiency Coding and Media Delivery in Heterogeneous Environments—Part 12: Image File Format”, ISO/IEC 23008-12, [online], Dec. 2017, <URL: https://standards.iso.org/ittf/PubliclyAvailableS… [cited by applicant]
“Information Technology—High Efficiency Coding and Media Delivery in Heterogeneous Environments—Part 12: Image File Format”, ISO/IEC 23008-12:2017(E), First edition, Dec. 2017, ISO/IEC, pp. 39-50. (cited in JP2021-14285… [cited by applicant]
“Information Technology—High Efficiency Coding and Media Delivery in Heterogeneous Environments—Part 2: High Efficiency Video Coding”, ISO/IEC 23008-2:2020(E), Fourth edition, Aug. 2020, ISO/IEC, pp. 447-449. (cited in … [cited by applicant]
Vadakital et al. “HEVC Still Image file format (MPEG-H Part 12),” ISO/IEC JTC1/SC29/WG11 Coding of Moving Pictures and Audio, MPEG 2013 W14148, Jan. 17, 2014. [online]<URL: https://mpeg.chiariglione.org/standards/mpeg-h… [cited by applicant]
“3rd Generation Partnership Project; Technical Specification Group TS G SA; 5G Video Codec Characteristics (Release 17)”, 3GPP TR26.955 3GPP Aug. 8, 2020, V0.3.0 (cited in JP2021-142853 OA mailed Nov. 1, 2022). [cited by applicant]
Final Office Action from U.S. Appl. No. 17/464,251 dated Oct. 24, 2022. [cited by applicant]
https://www.iso.org/obp/ui/#iso:std:iso-iec: 14496:-15:ed-6:v1:en, “VVC Terms and Definitions,”. [cited by applicant]
Non-Final Office Action from U.S. Appl. No. 17/464,251 dated Apr. 27, 2023. [cited by applicant]
Aklouf et al. “Low Complexity Versatile Video Coding (VVC) for Low Bitrate Applications,” 2019 8th European Workshop on Visual Information Processing (EUVIP), Jan. 2, 2020, pp. 22-27. (cited in CN 202111028428.3 NOA mai… [cited by applicant]
Wang, Qiuyue. “Research on Technology of Inter-prediction and Rate-distortion Optimization in Video Coding,” A Master Thesis Submitted to University of Electronic Science and Technology of China, Jul. 15, 2020., whole d… [cited by applicant]
Zhang et al. “Video Composed of Image Sequence Using DES,” Computer Technology and Development, Jun. 2010, 20(6):209-216, whole document. (cited in CN 202111028428.3 NOA mailed Sep. 14, 2023). [cited by applicant]
Okubo et al.“H. 265 / HEVC textbook”, the first edition, Impress Corporation Japan, Oct. 21, 2013, 93-95, 192nd-197 pages, ISBN: 978-4-8443-3468-2. [cited by applicant]
Notice of Reasons for Refusal from Japanese Patent Application No. 2023-135957 dated Jun. 11, 2024. [cited by applicant]
Notice of Reasons for Refusal for Japanese Application No. 2021-142855, mailed Oct. 8, 2024, 11 pages. [cited by applicant]
Feng Rui et al.; The Constrution of Learning Social Networking Sites in the Perspective Connectionism; Journal of Distance Education; Issue 3; Apr. 2013; p. 10-16 (English Abstract on p. 16). [cited by applicant]
Hannuksela M.M., et al., “VVC Subpictures in ISO/IEC 14496-15,” ISO/IEC JTC 1/SC 29/WG 11, MPEG 2020/M52209, 129 MPEG Meeting, Brussels, BE, Jan. 13-17, 2020, 5 Pages, Retrieved on [Jan. 8, 2020]. [cited by applicant]
Pre-Appeal Report from Japanese Patent Application No. 2023-135104 dated May 23, 2025, 13 pages. [cited by applicant]
Document: JVET-S2001-vG, Bross, B., et al., “Versatile Video Coding (Draft 10),” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 19th Meeting: by teleconference, 22 Jun.-Jul. 1, 2020, 5… [cited by applicant]