IP Library › Granted Patent US 12,382,041
Granted Patent B2
US 12,382,041 · App. 17/953,045 · Granted Aug 5, 2025

Coded video processing using enhanced secondary transform

Inventors: Kai Zhang (San Diego, CA); Li Zhang (San Diego, CA); Tianliang Fu (Beijing, CN); Yue Wang (Beijing, CN); Siwei Ma (Beijing, CN)
Assignees: BEIJING BYTEDANCE NETWORK TECHNOLOGY CO., LTD.; BYTEDANCE INC.
H04N19/12H04N19/124H04N19/126H04N19/172H04N19/176H04N19/189H04N19/60H04N19/61H04N19/70
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,382,041
App. No.
17/953,045
Granted
Aug 5, 2025
Kind
B2
Abstract

A video processing method includes performing a conversion between a video block of a video and a bitstream of the video according to a rule. The rule specifies whether or how usage of a secondary transform within a video unit is indicated in the bitstream. The secondary transform is applied before quantization or after de-quantization.

Claims (51)

1. A method of processing video data, comprising:

performing a conversion between a video block of a video and a bitstream of the video according to a rule,

wherein the rule specifies whether an implicit indication manner or an explicit indication manner is applied for indicating a usage of a secondary transform for the video block within a video unit is indicated in the bitstream,

wherein the secondary transform is applied before quantization or after de-quantization, and

wherein the secondary transform is applied between forward primary transform and quantization or between de-quantization and invert primary transform;

wherein whether the implicit indication manner or the explicit indication manner is applied for indicating the usage of the secondary transform is based on a coding mode of the video block;

wherein whether the implicit indication manner or the explicit indication manner is applied for indicating the usage of the secondary transform is determined based on whether a pulse-coded modulation (PCM) coding tool is applied to the video block; and

wherein in response to the PCM coding tool being applied to the video block, the implicit indication manner of not using the secondary transform is applied to the video block.

2. The method of claim 1 , wherein the video unit comprises a video picture of the video.

3. The method of claim 1 , wherein the video unit comprises a video sequence of the video.

4. The method of claim 1 , wherein the secondary transform comprises an enhanced secondary transform.

5. The method of claim 1 , wherein whether the implicit indication manner or the explicit indication manner is applied for indicating the usage of the secondary transform is determined based on whether the video block is coded with a derived tree partition.

6. The method of claim 1 , wherein multiple ways of implicit indication manners are utilized to video blocks within the video unit.

7. The method of claim 1 , wherein in response to a non-intra prediction coding mode being applied to the video block, the implicit indication manner of not using the secondary transform is applied to the video block.

8. The method of claim 1 , wherein a syntax element is used for the explicit indication manner of the usage of the secondary transform.

9. The method of claim 1 , wherein the conversion comprises encoding the video into the bitstream.

10. The method of claim 1 , wherein the conversion comprises decoding the video from the bitstream.

11. An apparatus for processing video data comprising a processor and a non-transitory memory with instructions thereon, wherein the instructions upon execution by the processor, cause the processor to:

perform a conversion between a video block of a video and a bitstream of the video according to a rule,

wherein the rule specifies whether an implicit indication manner or an explicit indication manner is applied for indicating a usage of a secondary transform for the video block within a video unit is indicated in the bitstream,

wherein the secondary transform is applied before quantization or after de-quantization, and

wherein the secondary transform is applied between forward primary transform and quantization or between de-quantization and invert primary transform;

wherein whether the implicit indication manner or the explicit indication manner is applied for indicating the usage of the secondary transform is based on a coding mode of the video block;

wherein whether the implicit indication manner or the explicit indication manner is applied for indicating the usage of the secondary transform is determined based on whether a pulse-coded modulation (PCM) coding tool is applied to the video block; and

wherein in response to the PCM coding tool being applied to the video block, the implicit indication manner of not using the secondary transform is applied to the video block.

12. The apparatus of claim 11 , wherein the video unit comprises a video sequence of the video.

13. A non-transitory computer-readable storage medium storing instructions that cause a processor to:

perform a conversion between a video block of a video and a bitstream of the video according to a rule,

wherein the rule specifies whether an implicit indication manner or an explicit indication manner is applied for indicating a usage of a secondary transform for the video block within a video unit is indicated in the bitstream,

wherein the secondary transform is applied before quantization or after de-quantization, and

wherein the secondary transform is applied between forward primary transform and quantization or between de-quantization and invert primary transform;

wherein whether the implicit indication manner or the explicit indication manner is applied for indicating the usage of the secondary transform is based on a coding mode of the video block;

wherein whether the implicit indication manner or the explicit indication manner is applied for indicating the usage of the secondary transform is determined based on whether a pulse-coded modulation (PCM) coding tool is applied to the video block; and

wherein in response to the PCM coding tool being applied to the video block, the implicit indication manner of not using the secondary transform is applied to the video block.

14. The non-transitory computer-readable storage medium of claim 13 , wherein whether the implicit indication manner or the explicit indication manner is applied for indicating the usage of the secondary transform is determined based on whether the video block is coded with a derived tree partition; or

wherein whether the implicit indication manner or the explicit indication manner is applied for indicating the usage of the secondary transform is determined based on whether a pulse-coded modulation (PCM) coding tool is applied to the video block, wherein in response to the PCM coding tool being applied to the video block, the implicit indication manner of not using the secondary transform is applied to the video block; or

wherein in response to a non-intra prediction coding mode being applied to the video block, the implicit indication manner of not using the secondary transform is applied to the video block.

15. The non-transitory computer-readable storage medium of claim 13 , wherein a syntax element is used for the explicit indication manner of the usage of the secondary transform.

16. A method for storing a bitstream of a video, comprising:

generating the bitstream of the video from a video block of the video according to a rule; and

storing the bitstream in a non-transitory computer-readable recording medium,

wherein the rule specifies whether an implicit indication manner or an explicit indication manner is applied for indicating a usage of a secondary transform for the video block within a video unit is indicated in the bitstream,

wherein the secondary transform is applied before quantization or after de-quantization, and

wherein the secondary transform is applied between forward primary transform and quantization or between de-quantization and invert primary transform;

wherein whether the implicit indication manner or the explicit indication manner is applied for indicating the usage of the secondary transform is based on a coding mode of the video block;

wherein whether the implicit indication manner or the explicit indication manner is applied for indicating the usage of the secondary transform is determined based on whether a pulse-coded modulation (PCM) coding tool is applied to the video block; and

wherein in response to the PCM coding tool being applied to the video block, the implicit indication manner of not using the secondary transform is applied to the video block.

17. The method of claim 16 , wherein whether the implicit indication manner or the explicit indication manner is applied for indicating the usage of the secondary transform is determined based on whether the video block is coded with a derived tree partition; or

wherein whether the implicit indication manner or the explicit indication manner is applied for indicating the usage of the secondary transform is determined based on whether a pulse-coded modulation (PCM) coding tool is applied to the video block, wherein in response to the PCM coding tool being applied to the video block, the implicit indication manner of not using the secondary transform is applied to the video block; or

wherein in response to a non-intra prediction coding mode being applied to the video block, the implicit indication manner of not using the secondary transform is applied to the video block.

18. The method of claim 16 , wherein a syntax element is used for the explicit indication manner of the usage of the secondary transform.

Assignments (4)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Feb 16, 2023
From: ZHANG, KAI; ZHANG, LI
To: BYTEDANCE INC.
Reel/Frame 062721/0229 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Feb 16, 2023
From: FU, TIANLIANG; MA, SIWEI
To: BEIJING BYTEDANCE NETWORK TECHNOLOGY CO., LTD.
Reel/Frame 062721/0328 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Feb 16, 2023
From: WANG, YUE
To: BEIJING ZITIAO NETWORK TECHNOLOGY CO., LTD.
Reel/Frame 062721/0491 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Feb 16, 2023
From: BEIJING ZITIAO NETWORK TECHNOLOGY CO., LTD.
To: BEIJING BYTEDANCE NETWORK TECHNOLOGY CO., LTD.
Reel/Frame 062721/0560 →
Priority Claims (1)
WO PCT/CN2020/081174 · Mar 25, 2020 · international
Continuity (2)
Continuation PCTCN2021082962 · Mar 25, 2021
Related Publication 20230037443A1 · Feb 9, 2023
References Cited (122)
US 5822004A · Crocitti et al. · 1998 [cited by applicant]
US 6389072B1 · Tzou et al. · 2002 [cited by applicant]
US 10666976B2 · Huang et al. · 2020 [cited by applicant]
US 11647229B2 · Zhang · 2023 [cited by applicant]
US 20020044605A1 · Nakamura · 2002 [cited by applicant]
US 20040264571A1 · Zhang et al. · 2004 [cited by applicant]
US 20130003856A1 · Saxena et al. · 2013 [cited by applicant]
US 20140254676A1 · Jiang et al. · 2014 [cited by applicant]
US 20160050422A1 · Rosewarne et al. · 2016 [cited by applicant]
US 20170094313A1 · Zhao et al. · 2017 [cited by applicant]
US 20180041776A1 · Kim et al. · 2018 [cited by applicant]
US 20180103252A1 · Hseih et al. · 2018 [cited by applicant]
US 20180302631A1 · Chiang et al. · 2018 [cited by applicant]
US 20180338143A1 · Fracastoro et al. · 2018 [cited by applicant]
US 20180367814A1 · Seregin et al. · 2018 [cited by applicant]
US 20190028701A1 · Yu et al. · 2019 [cited by applicant]
US 20190141334A1 · Lim · 2019 [cited by examiner]
US 20190356915A1 · Jang et al. · 2019 [cited by applicant]
US 20200288134A1 · Lim · 2020 [cited by examiner]
US 20200288172A1 · Huang et al. · 2020 [cited by applicant]
US 20200322617A1 · Zhao et al. · 2020 [cited by applicant]
US 20210195222A1 · Egilmez · 2021 [cited by examiner]
US 20210321134A1 · Koo et al. · 2021 [cited by applicant]
US 20210352326A1 · Lim · 2021 [cited by examiner]
US 20220046281A1 · Nam · 2022 [cited by examiner]
US 20220109876A1 · Zhang et al. · 2022 [cited by applicant]
US 20220132134A1 · Koo · 2022 [cited by examiner]
US 20220150498A1 · Zhang · 2022 [cited by examiner]
US 20220159300A1 · Chiang · 2022 [cited by examiner]
US 20220182675A1 · Zhang et al. · 2022 [cited by applicant]
US 20220201335A1 · Chiang et al. · 2022 [cited by applicant]
US 20220248055A1 · Koo · 2022 [cited by examiner]
US 20220264101A1 · Koo · 2022 [cited by examiner]
US 20220312013A1 · Koo · 2022 [cited by examiner]
US 20220329786A1 · Huo · 2022 [cited by examiner]
US 20220329861A1 · Huo · 2022 [cited by examiner]
US 20220329862A1 · Huo · 2022 [cited by examiner]
US 20220345744A1 · LeLeannec · 2022 [cited by examiner]
US 20220360785A1 · Huo · 2022 [cited by examiner]
US 20220385912A1 · Koo · 2022 [cited by examiner]
US 20220385946A1 · Chiang · 2022 [cited by examiner]
US 20230078566A1 · Koo · 2023 [cited by examiner]
US 20230130131A1 · Chiang · 2023 [cited by examiner]
US 20230396805A1 · Le Leannec · 2023 [cited by examiner]
US 20240007674A1 · Koo · 2024 [cited by examiner]
CN 104025589A · 2014 [cited by applicant]
CN 105516730A · 2016 [cited by applicant]
CN 108141594A · 2018 [cited by applicant]
CN 108141596A · 2018 [cited by applicant]
CN 108141597A · 2018 [cited by applicant]
CN 108322745A · 2018 [cited by applicant]
CN 108632611A · 2018 [cited by applicant]
CN 108712649A · 2018 [cited by applicant]
CN 109076222A · 2018 [cited by applicant]
CN 109076223A · 2018 [cited by applicant]
CN 109076226A · 2018 [cited by applicant]
CN 109076230A · 2018 [cited by applicant]
CN 109076242A · 2018 [cited by applicant]
CN 109076243A · 2018 [cited by applicant]
CN 109644269A · 2019 [cited by applicant]
CN 110677675A · 2020 [cited by applicant]
CN 110636313B · 2022 [cited by examiner]
EP 3349451A1 · 2018 [cited by applicant]
EP 3506634A4 · 2019 [cited by applicant]
JP 7509944B2 · 2024 [cited by applicant]
KR 20180041578A · 2018 [cited by applicant]
KR 20210120802A1 · 2021 [cited by examiner]
KR 2321394B1 · 2021 [cited by examiner]
KR 102727219B1 · 2024 [cited by applicant]
TW 201902219A · 2019 [cited by applicant]
WO 2017195555A1 · 2017 [cited by applicant]
WO 2017195666A1 · 2017 [cited by applicant]
WO 2018037737A1 · 2018 [cited by applicant]
WO 2018128323A1 · 2018 [cited by applicant]
WO 2018166429A1 · 2018 [cited by applicant]
WO 2018174402A1 · 2018 [cited by applicant]
WO 2019006148A1 · 2019 [cited by applicant]
WO 2019022099A1 · 2019 [cited by applicant]
WO 2020046092A1 · 2020 [cited by applicant]
WO 2020049446A1 · 2020 [cited by applicant]
WO WO2021194052A1 · 2021 [cited by examiner]
Machine translation of “From HEVC to WC: Evolution of transformation technology (2)—Secondary transform” (Year: 2019). [cited by examiner]
Columbia Tristar Home Video, optical disc storing video bitstream of motion picture “Anatomy of a Murder” (Year: 2000). [cited by examiner]
CN-110636313-B machine translation (Year: 2022). [cited by examiner]
KR-10-2321394-B1 machine translation (Year: 2021). [cited by examiner]
KR-2021-0120802-A1 machine translation (Year: 2021). [cited by examiner]
WO-2021-194052-A1 machine translation (Year: 2021). [cited by examiner]
Document: JVET-N1001-v2, Bross, B., “Versatile Video Coding (Draft 5),” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 14th Meeting: Geneva, CH, Mar. 19-27, 2019, 361 pages. [cited by applicant]
Bossen, F., vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM/tags/VTM-4.0, VTM version 4.0, Dec. 7, 2022, 2 pages. [cited by applicant]
Document: JVET-M0102-v5, De-Luxan-Hernandez, S., et al., “CE3: Intra Sub-Partitions Coding Mode (Tests 1.1.1 and 1.1.2),” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 13th Meeting: M… [cited by applicant]
Document: JVET-M0057, Abdoli, M., et al., CE8: BDPCM with horizontal/vertical predictor and independently decodable areas (test 8.3.1b), Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 … [cited by applicant]
Document: JVET-N0413, Karczewicz, M., et al., “CE8-related: Quantized residual BDPCM,” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 14th Meeting: Geneva, CH, Mar. 19-27, 2019, 6 page… [cited by applicant]
Document: JVET-N0193, Koo, M., et al., “CE6: Reduced Secondary Transform (RST) (CE6-3.1),” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 14th Meeting: Geneva, CH, Mar. 19-27, 2019, 19… [cited by applicant]
Document: JVET-K0099, Salehifar, M., et al., “CE 6.2.6: Reduced Secondary Transform (RST),” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 11th Meeting: Ljubljana, SI, Jul. 10-18, 2018… [cited by applicant]
Document: JVET-L0133, Koo, M., et al., “CE 6-2.1: Reduced Secondary Transform (RST),” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 12th Meeting: Macao, CN, Oct. 3-12, 2018, 8 pages. [cited by applicant]
Document: JVET-N0217, Pfaff, J., et al., “CE3: Affine linear weighted intra prediction (CE3-4.1, CE3-4.2),” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 14th Meeting: Geneva, CH, Mar… [cited by applicant]
Document: JVET-K0102-v1, Zhang, K., et al., “CE4-related: Interweaved Prediction for Affine Motion Compensation,” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 11th Meeting: Ljubljana… [cited by applicant]
Document: JVET-N0555-v3, Siekmann, M., et al., “CE6-related: Simplification of the Reduced Secondary Transform,” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 14th Meeting: Geneva, CH… [cited by applicant]
Document: JVET-M0292, Koo, M., et al., “CE6: Reduced Secondary Transform (RST) (test 6.5.1),” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 13th Meeting: Marrakech, MA, Jan. 9-18, 201… [cited by applicant]
Document: JVET-B0059, Zhao, X., et al., “TU-level non-separable secondary transform,” Joint Video Exploration Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 2nd Meeting: San Diego, USA, Feb. 20-26, 2016, … [cited by applicant]
“From HEVC to VVC Evolution of Transform Technology (2)—Secondary Transform,” retrieved from the internet: https://cloud.tencent.com/developer/article/1427150, May 15, 2019, 6 pages. [cited by applicant]
Koo, M., et al., “Low Frequency Non-Separable Transform (LFNST),” 2019 Picture Coding Symposium (PCS), Nov. 12-15, 2019, 5 pages. [cited by applicant]
Zhao, X., et al., “NSST: Non-Separable secondary transforms for next generation video coding,” 2016 Picture Coding Symposium (PCS), Dec. 2016, 5 pages. [cited by applicant]
Document: JVET-P0379-v1, Fan, K., et al., “Non-CE6: A unified zero-out range for 4x4 LFNST,” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 16th Meeting: Geneva, CH, Oct. 1-11, 2019, 8… [cited by applicant]
Document: JVET-N1002-v2, Chen, J., et al., “Algorithm description for Versatile Video Coding and Test Model 5 (VTM 5),” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 14th Meeting: Gen… [cited by applicant]
Foreign Communication From a Related Counterpart Application, PCT Application No. PCT/CN2020/086421, English Translation of International Search Report dated Jul. 23, 2020, 11 pages. [cited by applicant]
Non-Final Office Action dated Dec. 29, 2021, 22 pages, U.S. Appl. No. 17/406,260, filed Aug. 19, 2021. [cited by applicant]
Foreign Communication From a Related Counterpart Application, PCT Application No. PCT/CN2020/086444, English Translation of International Search Report dated Aug. 11, 2020, 11 pages. [cited by applicant]
Foreign Communication From a Related Counterpart Application, European Application No. 20795742.4, Extended European Search Report dated May 3, 2022, 9 pages. [cited by applicant]
Non-Final Office Action dated Dec. 13, 2021, 21 pages, U.S. Appl. No. 17/406,242, filed Aug. 19, 2021. [cited by applicant]
Foreign Communication From a Related Counterpart Application, PCT Application No. PCT/CN2020/086458, English Translation of International Search Report dated Jul. 6, 2020, 9 pages. [cited by applicant]
Foreign Communication From a Related Counterpart Application, PCT Application No. PCT/CN2020/096046, English Translation of International Search Report dated Sep. 24, 2020, 11 pages. [cited by applicant]
Foreign Communication From a Related Counterpart Application, PCT Application No. PCT/CN2020/133273, English Translation of International Search Report dated Feb. 18, 2021, 10 pages. [cited by applicant]
Foreign Communication From a Related Counterpart Application, PCT Application No. PCT/CN2021/082962, English Translation of International Search Report dated Jul. 1, 2021, 10 pages. [cited by applicant]
Bossen F., “VTM-4.0,” 2 Pages, Retrieved from URL: vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM/tags/VTM-4.0, retrieved Jun. 29, 2024. [cited by applicant]
JVET-N0555-v1, Siekmann M., et al., “CE6-Related: Simplification of the Reduced Secondary Transform,” Fraunhofer HHI, Mar. 24, 2019, 10 Pages. [cited by applicant]
Non-Final Office Action for U.S. Appl. No. 17/406,242, mailed Aug. 4, 2022, 18 Pages. [cited by applicant]
Siekmann, M., et al., “CE6—related: Simplification of the Reduced Secondary Transform,” JVET of ITU-T and ISO/IEC, JVET-N0555.pptx in JVET-N0555 version 4 (Mar. 24, 2019), 10 pages. [cited by applicant]
Document: JCTVC-E380, Alshina, E., et al., “CE7: Experimental results of ROT by Samsung,” Joint Collaborative Team on Video Coding (JCT-VC) of ITU-T SG16 WP3 and ISO/IEC JTC1/SC29/WG11 5th Meeting: Geneva, CH, Mar. 16-2… [cited by applicant]
Document: JCTVC-I0408, Lan, C., et al., “Intra transform skipping,” Joint Collaborative Team on Video Coding (JCT-VC) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 9th Meeting: Geneva, CH, Apr. 27-May 7, 2012, 6 pag… [cited by applicant]
Non-Final Office Action from U.S. Appl. No. 18/193,131 dated Jan. 24, 2024, 28 pages. [cited by applicant]
Chinese Office Action from Chinese Patent Application No. 202180024692.1 dated Dec. 10, 2024, 51 pages. [cited by applicant]