IP Library Granted Patent US 12,615,366
Granted Patent B2
US 12,615,366 · App. 18/507,838 · Granted Apr 28, 2026

Low frequency non-separable transform signaling in video coding

Inventors: Ye-kui Wang (San Diego, CA); Li Zhang (San Diego, CA); Kai Zhang (San Diego, CA)
Assignee: Bytedance Inc.
H04N19/12H04N19/176H04N19/18H04N19/60H04N19/70
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,615,366
App. No.
18/507,838
Granted
Apr 28, 2026
Kind
B2
Abstract

Several techniques for video encoding and video decoding are described. A method includes performing a conversion between a video block of a video and a bitstream of the video, wherein the video block uses a low frequency non-separable transform for the conversion, wherein the bitstream conforms to a format rule that specifies that a syntax element is included at a syntax level for the bitstream, wherein the syntax element is indicative of whether use of a scaling matrix which is derived from a reference scaling list is enabled for the video block, and wherein the syntax level is a sequence level, a picture level, or a slice level.

Claims (57)

1 . A method of video processing, comprising:

performing a conversion between a video block of a video and a bitstream of the video,

wherein the video block uses a low frequency non-separable transform for the conversion,

wherein the bitstream conforms to a format rule that specifies that a first syntax element is included at a sequence level for the bitstream,

wherein the first syntax element indicates whether use of explicit scaling matrices, for blocks coded with the low frequency non-separable transform, is enabled, and

wherein when present, the first syntax element is included in a sequence parameter set (SPS), and the first syntax element being included in the SPS is determined based on the low frequency non-separable transform being enabled,

wherein the method further comprises:

determining to apply a default scaling matrix to the video block, based on a value of the first syntax element indicating that use of the explicit scaling matrices for blocks coded with the low frequency non-separable transform is disabled and the low frequency non-separable transform being enabled for the video block.

2 . The method of claim 1 , wherein when a value of a second syntax element included in the SPS is equal to 1, the first syntax element is included in the SPS, and

wherein the value of the second syntax element equal to 1 indicates that the low frequency non-separable transform is enabled, and the second syntax element is sps_lfnst_enabled_flag.

3 . The method of claim 1 , wherein the value of the first syntax element indicating that use of the explicit scaling matrices for blocks coded with the low frequency non-separable transform is disabled is equal to 1.

4 . The method of claim 3 , wherein scaling factors of the default scaling matrix are equal to 16.

5 . The method of claim 1 , wherein the format rule further specifies that a third syntax element is included in the bitstream at a video region level to control a scaling process applied to transform coefficients or non-transform coefficients, and

wherein a video region is smaller than a video picture of the video.

6 . The method of claim 5 , wherein the format rule specifies that the video region is a slice, and the third syntax element is included in a slice header.

7 . The method of claim 1 , wherein the performing the conversion comprises encoding the video into the bitstream.

8 . The method of claim 1 , wherein the performing the conversion comprises decoding the video from the bitstream.

9 . An apparatus for processing video data comprising a processor and a non-transitory memory with instructions thereon, wherein the instructions upon execution by the processor, cause the processor to:

perform a conversion between a video block of a video and a bitstream of the video,

wherein the video block uses a low frequency non-separable transform for the conversion,

wherein the bitstream conforms to a format rule that specifies that a first syntax element is included at a sequence level for the bitstream,

wherein the first syntax element indicates whether use of explicit scaling matrices, for blocks coded with the low frequency non-separable transform, is enabled, and

wherein when present, the first syntax element is included in a sequence parameter set (SPS), and the first syntax element being included in the SPS is determined based on the low frequency non-separable transform being enabled,

wherein the processor is further caused to:

determine to apply a default scaling matrix to the video block, based on a value of the first syntax element indicating that use of the explicit scaling matrices for blocks coded with the low frequency non-separable transform is disabled and the low frequency non-separable transform being enabled for the video block.

10 . The apparatus of claim 9 , wherein when a value of a second syntax element included in the SPS is equal to 1, the first syntax element is included in the SPS, and

wherein the value of the second syntax element equal to 1 indicates that the low frequency non-separable transform is enabled, and the second syntax element is sps_lfnst_enabled_flag.

11 . A non-transitory computer-readable storage medium storing instructions that cause a processor to:

perform a conversion between a video block of a video and a bitstream of the video,

wherein the video block uses a low frequency non-separable transform for the conversion,

wherein the bitstream conforms to a format rule that specifies that a first syntax element is included at a sequence level for the bitstream,

wherein the first syntax element indicates whether use of explicit scaling matrices, for blocks coded with the low frequency non-separable transform, is enabled, and

wherein when present, the first syntax element is included in a sequence parameter set (SPS), and the first syntax element being included in the SPS is determined based on the low frequency non-separable transform being enabled,

wherein the processor is further caused to:

determine to apply a default scaling matrix to the video block, based on a value of the first syntax element indicating that use of the explicit scaling matrices for blocks coded with the low frequency non-separable transform is disabled and the low frequency non-separable transform being enabled for the video block.

12 . The non-transitory computer-readable storage medium of claim 11 , wherein the instructions further cause the processor to:

wherein the value of the first syntax element indicating that use of the explicit scaling matrices for blocks coded with the low frequency non-separable transform is disabled is equal to 1.

13 . A method for storing a bitstream of a video, comprising:

generating the bitstream of the video from a video block of the video, and

storing the bitstream in a non-transitory computer-readable recording medium,

wherein the video block uses a low frequency non-separable transform for the generation,

wherein the bitstream conforms to a format rule that specifies that a first syntax element is included at a sequence level for the bitstream,

wherein the first syntax element indicates whether use of explicit scaling matrices, for blocks coded with the low frequency non-separable transform, is enabled, and

wherein when present, the first syntax element is included in a sequence parameter set (SPS), and the first syntax element being included in the SPS is determined based on the low frequency non-separable transform being enabled,

wherein the method further comprises:

determining to apply a default scaling matrix to the video block, based on a value of the first syntax element indicating that use of the explicit scaling matrices for blocks coded with the low frequency non-separable transform is disabled and the low frequency non-separable transform being enabled for the video block.

14 . The method of claim 13 , wherein the method further comprises:

wherein the value of the first syntax element indicating that use of the explicit scaling matrices for blocks coded with the low frequency non-separable transform is disabled is equal to 1, which indicates that use of the explicit scaling matrices for blocks coded with the low frequency non-separable transform is disabled and the low frequency non-separable transform is enabled for the video block, the default scaling matrix is applied to the video block.

15 . The apparatus of claim 9 , wherein the value of the first syntax element indicating that use of the explicit scaling matrices for blocks coded with the low frequency non-separable transform is disabled is equal to 1.

16 . The apparatus of claim 15 , wherein scaling factors of the default scaling matrix are equal to 16.

17 . The apparatus of claim 9 , wherein the format rule further specifies that a third syntax element is included in the bitstream at a video region level to control a scaling process applied to transform coefficients or non-transform coefficients, and

wherein a video region is smaller than a video picture of the video.

18 . The apparatus of claim 17 , wherein the format rule specifies that the video region is a slice, and the third syntax element is included in a slice header.

19 . The non-transitory computer-readable storage medium of claim 11 , wherein when a value of a second syntax element included in the SPS is equal to 1, the first syntax element is included in the SPS, and

wherein the value of the second syntax element equal to 1 indicates that the low frequency non-separable transform is enabled, and the second syntax element is sps_lfnst_enabled_flag.

20 . The non-transitory computer-readable storage medium of claim 11 , wherein the format rule further specifies that a third syntax element is included in the bitstream at a video region level to control a scaling process applied to transform coefficients or non-transform coefficients, and

wherein a video region is smaller than a video picture of the video.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 5, 2023
From: WANG, YE-KUI; ZHANG, LI; ZHANG, KAI
To: BYTEDANCE INC.
Reel/Frame 065769/0172 →
Continuity (4)
Continuation 17901270 · Sep 1, 2022
Continuation PCTUS2021020618 · Mar 3, 2021
Provisional Application 62984658 · Mar 3, 2020
Related Publication 20240121389A1 · Apr 11, 2024
References Cited (106)
US 9491456B2 · Wang · 2016 [cited by applicant]
US 10511843B2 · Fu · 2019 [cited by applicant]
US 11197006B2 · Poirier · 2021 [cited by applicant]
US 11297310B2 · Yoo · 2022 [cited by applicant]
US 11297317B2 · Han · 2022 [cited by applicant]
US 11477454B2 · Paluri · 2022 [cited by applicant]
US 11523108B2 · Zhang · 2022 [cited by applicant]
US 11533500B2 · Paluri · 2022 [cited by applicant]
US 11539950B2 · Zhang · 2022 [cited by applicant]
US 11601655B2 · Chen · 2023 [cited by applicant]
US 11856234B2 · Wang · 2023 [cited by applicant]
US 20120207227A1 · Tsai · 2012 [cited by applicant]
US 20130202051A1 · Zhou · 2013 [cited by applicant]
US 20130343465A1 · Chen · 2013 [cited by applicant]
US 20150264404A1 · Hannuksela · 2015 [cited by applicant]
US 20150358631A1 · Zhang · 2015 [cited by applicant]
US 20160353110A1 · Zhang · 2016 [cited by applicant]
US 20160366437A1 · Zhou · 2016 [cited by applicant]
US 20170238014A1 · Said · 2017 [cited by applicant]
US 20170238019A1 · Said · 2017 [cited by applicant]
US 20180103252A1 · Hsieh · 2018 [cited by applicant]
US 20180270480A1 · Zhang · 2018 [cited by applicant]
US 20180376126A1 · Hannuksela · 2018 [cited by applicant]
US 20190182495A1 · Bruns · 2019 [cited by applicant]
US 20190246143A1 · Zhang · 2019 [cited by applicant]
US 20190253624A1 · Kim · 2019 [cited by applicant]
US 20190387241A1 · Kim · 2019 [cited by applicant]
US 20200260070A1 · Yoo · 2020 [cited by applicant]
US 20200322623A1 · Chiang · 2020 [cited by applicant]
US 20200389671A1 · Zhao · 2020 [cited by applicant]
US 20200396475A1 · Furht · 2020 [cited by applicant]
US 20210092408A1 · Ramasubramonian · 2021 [cited by applicant]
US 20210136415A1 · Hashimoto · 2021 [cited by applicant]
US 20210195186A1 · Wu · 2021 [cited by applicant]
US 20210218965A1 · Li · 2021 [cited by applicant]
US 20210409779A1 · Li · 2021 [cited by applicant]
US 20220039430A1 · Blase · 2022 [cited by applicant]
US 20220394281A1 · Zhang · 2022 [cited by applicant]
US 20220394301A1 · Deshpande · 2022 [cited by applicant]
US 20230057736A1 · Hendry · 2023 [cited by applicant]
US 20230060709A1 · Ouedraogo · 2023 [cited by applicant]
CN 101115201A · 2008 [cited by applicant]
CN 103988437A · 2014 [cited by applicant]
CN 106464891A · 2017 [cited by applicant]
CN 106464934A · 2017 [cited by applicant]
CN 110662053A · 2020 [cited by applicant]
CN 115211125B · 2025 [cited by applicant]
EP 3020202A1 · 2016 [cited by applicant]
EP 4032304A1 · 2022 [cited by applicant]
GB 2584295A · 2020 [cited by applicant]
GB 2590636A · 2021 [cited by applicant]
IN 548726 · 2024 [cited by applicant]
JP 2013236358A · 2013 [cited by applicant]
JP 7623077B2 · 2025 [cited by applicant]
KR 102706671B1 · 2024 [cited by applicant]
WO 2013154869A1 · 2013 [cited by applicant]
WO 2018175409A1 · 2018 [cited by applicant]
WO 2019073112A1 · 2019 [cited by applicant]
WO 2019167849A1 · 2019 [cited by applicant]
WO 2019185883A1 · 2019 [cited by applicant]
WO 2019190907A1 · 2019 [cited by applicant]
WO 2020011796A1 · 2020 [cited by applicant]
WO 2020035827A1 · 2020 [cited by applicant]
Paluri, Seethal etal., (hereinafter Paluri'299) AHG17: “APS support for default and user defined scaling matrices” (JVET-O0299): Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTCI/SC 29/WG 11, 15th Mee… [cited by examiner]
Document: JVET-Q2001-Vc, Bross, B., et al., “Versatile Video Coding (Draft 8),” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 17th Meeting: Brussels, BE, Jan. 7-17, 2020, 508 pages. [cited by applicant]
“Series H: Audiovisual and Multimedia Systems Infrastructure of audiovisual services—Coding of moving video High efficiency video coding,” Recommendation ITU-T H.265, Feb. 2018, 692 pages. [cited by applicant]
Document: JVET-G1001-v1, Chen, J., et al., “Algorithm Description of Joint Exploration Test Model 7 (JEM 7),” Joint Video Exploration Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 7th Meeting: Torino, IT… [cited by applicant]
Document: JVET-Q2001-v3, Bross, B., et al., “Versatile Video Coding (Draft 8),” Joint Video Experts Team (JVET) of ITU-TSG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 17th Meeting: Brussels, BE, Jan. 7-17, 2020, 39 pages. [cited by applicant]
Document: JVET-Q2002-v3, Chen, J., et al., “Algorithm description for Versatile Video Coding and Test Model 8 (VTM 8),” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 17th Meeting: Bru… [cited by applicant]
Li, X., “VVC VTM reference software,” Retrieved from the internet: https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM.git, VTM software, Nov. 22, 2022, 3 pages. [cited by applicant]
Document: JCTVC-AC1005-v2, Boyce, J., et al., “HEVC Additional Supplemental Enhancement Information (Draft 4),” Joint Collaborative Team on Video Coding (JCT-VC) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 29th Me… [cited by applicant]
Sulllivan et al. “Meeting Report of the 17th Meeting of the Joint Video Experts Team (JVET)”. In www.itu.int [online]. [shortened pp. 1-7.252-270]. published Feb. 17, 2020 [retrieved on Apr. 7, 2021]. Retrieved from the… [cited by applicant]
Aklouf, M., et al., “Low Complexity Versatile Video Coding (VVC) for Low Bitrate Applications,” 8th European Workshop on Visual Information Processing (EUVIP 2019), hal-02299789, Oct. 2019, 7 pages. [cited by applicant]
Document: JVET-Q_Notes_dF, Sullivan, G., et al. Meeting Report of the 17th Meeting of the Joint Video Experts Team (JVET), Brussels, BE, Jan. 7-17, 2020, 20 pages. [cited by applicant]
Koo, M., et al., “Low Frequency Non-Separable Transform (LFNST)” IEEE, 2019 Picture Coding Symposium (PCS, Nov. 12-15, 2019, 5 pages. [cited by applicant]
Document: JVET-M1001-v7, Bross, B., et al., “Versatile Video Coding; Draft 4,” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 13th Meeting: Marrakech, MA, Jan. 9-18, 2019, 300 pages. [cited by applicant]
Document: JVET-Q2001-vD, Bross, B., et al., “Versatile Video Coding (Draft 8),” Joint Video Experts Team (JVET) of ITU-TSG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 17th Meeting: Brussels, BE, Jan. 7-17, 2020, 509 pages. [cited by applicant]
Document: JVET-P0365-v2, Ramasubramonian, A., et al., “AHG15: Scaling matrices for LFNST-coded blocks,” Joint Video Experts Team (JVET) of ITU-TSG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 16th Meeting: Geneva, CH, Oct. 1-1… [cited by applicant]
Document: JVET-Q0346, Samuelsson, J., et al., “AHG9: On Slice Header,” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 17th Meeting: Brussels, BE Jan. 7-17, 2020, 2 pages. [cited by applicant]
Document: JVET-R0062-v1, Zhang, L., et al., “AHG12: A cleanup on uniform tile and rectangular slice partitioning,” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 18th Meeting: by telec… [cited by applicant]
Document: JVET-R0064-v1, Wang, Y., et al., “AHG9: Signalling of scaling list control,” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 18th Meeting: by teleconference, Apr. 15-24, 2020,… [cited by applicant]
Notice of Allowance dated Aug. 16, 2023, 19 pages, U.S. Appl. No. 17/894,832, filed Aug. 24, 2022. [cited by applicant]
Foreign Communication From A Related Counterpart Application, Indian Application No. 202227048235, Indian Office Action dated Dec. 8, 2022, 6 pages. [cited by applicant]
Non-Final Office Action dated Dec. 16, 2022, 23 pages, U.S. Appl. No. 17/893,500, filed Aug. 23, 2022. [cited by applicant]
Non-Final Office Action dated Jan. 3, 2023, 22 pages, U.S. Appl. No. 17/893,481, filed Aug. 23, 2022. [cited by applicant]
Non-Final Office Action dated Feb. 17, 2023, 17 pages, U.S. Appl. No. 17/894,888, filed Aug. 24, 2022. [cited by applicant]
Foreign Communication From A Related Counterpart Application, European Application No. 21760718.3, Extended European Search Report dated Feb. 27, 2023, 10 pages. [cited by applicant]
Foreign Communication From A Related Counterpart Application, European Application No. 21764233.9, Extended European Search Report dated Mar. 22, 2023, 9 pages. [cited by applicant]
Foreign Communication From A Related Counterpart Application, PCT Application No. PCT/US2021/019216, International Search Report dated May 7, 2021, 10 pages. [cited by applicant]
Foreign Communication From A Related Counterpart Application, PCT Application No. PCT/US2021/019217, International Search Report dated May 11, 2021, 16 pages. [cited by applicant]
Foreign Communication From A Related Counterpart Application, PCT Application No. PCT/US2021/019228, International Search Report dated May 7, 2021, 9 pages. [cited by applicant]
Foreign Communication From A Related Counterpart Application, PCT Application No. PCT/US2021/019235, International Search Report dated May 17, 2021, 10 pages. [cited by applicant]
Foreign Communication From A Related Counterpart Application, PCT Application No. PCT/US2021/020615, International Search Report dated May 20, 2021, 15 pages. [cited by applicant]
Foreign Communication From A Related Counterpart Application, PCT Application No. PCT/US2021/020618, International Search Report dated Jul. 14, 2021, 22 pages. [cited by applicant]
Final Office Action dated Jul. 7, 2023, 19 pages, U.S. Appl. No. 17/901,270, filed Sep. 1, 2022. [cited by applicant]
Non Final Office Action dated Mar. 28, 2023, 31 pages, U.S. Appl. No. 17/901,270, filed Sep. 1, 2022. [cited by applicant]
Notice of Allowance dated Nov. 9, 2023, 14 pages, U.S. Appl. No. 17/901,270, filed Sep. 1, 2022. [cited by applicant]
Document: JVET-Q0119-v1, Wang, Y.K., et al., “AHG12: Cleanups on signalling of subpictures, tiles, and rectangular slices,” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 17th Meeting:… [cited by applicant]
Document: JVET-Q0787-v2, Suehring, K., et al., “AHG9: Subpicture location signalling bugfix,” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 17th Meeting: Brussels, BE, Jan. 7-17, 2020… [cited by applicant]
Non-Final Office Action from U.S. Appl. No. 18/516,507 dated Jul. 18, 2024, 23 pages. [cited by applicant]
Chinese Notice of Allowance from Chinese Patent Application No. 202180019098.3 dated Jul. 17, 2024, 6 pages. [cited by applicant]
Korean Notice of Allowance from Korean Patent Application No. 10-2022-7027814 dated Jun. 11, 2024, 8 pages. [cited by applicant]
Notice of Grant of patent right for invention from Japanese Patent Application No. 202180019067.8 dated Jan. 10, 2025, 4 pages. [cited by applicant]
Document: JVET-Q0044, Nishi, et al., “AHG9/AHG12: Simplification of slice index with subpictures”, Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 17th Meeting: Brussels, BE, Jan. 7-17,… [cited by applicant]
Document: JVET-Q0377, Ouedraogo, et al., “AHG12: On “tile-fraction” slices and signaling of slices per subpicture”, Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 17th Meeting: Brussel… [cited by applicant]
Singapore Office Action from Singapore Patent Application No. 11202252501P, dated Jul. 29, 2025, 10 pages. [cited by applicant]