IP Library › Granted Patent US 12,726,611
Granted Patent B2
US 12,726,611 · App. 18/882,254 · Granted Sep 1, 2026

Interaction between template matching-based local illumination compensation and a second merge list in video coding

Inventors: Yan Zhang (San Diego, CA); Vadim Seregin (San Diego, CA); Hongtao Wang (San Diego, CA); Zhi Zhang (Munich, DE); Chun-Chi Chen (San Diego, CA); Han Huang (San Diego, CA); Marta Karczewicz (San Diego, CA)
Assignee: QUALCOMM Incorporated
H04N19/105H04N19/109H04N19/157H04N19/176H04N19/44H04N19/463H04N19/513H04N19/573H04N19/70H04N19/52
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,726,611
App. No.
18/882,254
Granted
Sep 1, 2026
Kind
B2
Abstract

Example devices, methods, and computer-readable media for decoding video data are described. An example method includes determining to decode a current block of the video data using a merge mode. The example method includes generating a first merge list for the current block, wherein generating the first merge list comprises applying template matching to candidates of the first merge list. The example method includes generating, based on the first merge list, a second merge list. The example method includes decoding the current block using the merge mode and based on the first merge list or the second merge list.

Claims (35)

1 . A method of decoding video data, the method comprising:

determining to decode a current block of the video data using a merge mode;

generating a first merge list for the current block, wherein generating the first merge list comprises applying template matching to candidates of the first merge list;

generating, based on the first merge list for the current block, a second merge list for the current block;

obtaining a flag from a bitstream, the flag comprising a value indicative of the first merge list for the current block or the second merge list for the current block;

selecting, based on the value of the flag, a merge list from among the first merge list for the current block and the second merge list for the current block; and

decoding the current block using the merge mode and based on the selected merge list.

2 . The method of claim 1 , wherein generating the second merge list comprises, prior to applying template matching to the candidates of the first merge list, changing all of a plurality of local illumination compensation (LIC) flag values associated with the first merge list to opposite values for the second merge list.

3 . The method of claim 1 , wherein generating the second merge list comprises, after applying template matching to the candidates of the first merge list, changing all of a plurality of local illumination compensation (LIC) flag values associated with the first merge list to opposite values for the second merge list.

4 . The method of claim 1 , wherein generating the second merge list comprises, prior to applying template matching to the candidates of the first merge list, trimming candidates of the first merge list for the second merge list.

5 . The method of claim 1 , wherein generating the second merge list comprises, after applying template matching to the candidates of the first merge list, trimming candidates of the first merge list for the second merge list.

6 . The method of claim 1 , wherein generating the second merge list comprises:

determining that one or more candidates of the first merge list are bi-directional prediction merge candidates; and

for each of the bi-directional prediction merge candidates, changing a respective local illumination compensation (LIC) flag value associated with the first merge list to an opposite LIC flag value for the second merge list.

7 . The method of claim 1 , wherein generating the second merge list comprises:

determining whether a block size of the current block is greater than a threshold; and

based on a determination that the block size of the current block is greater than the threshold, determining either a) one or more local illumination compensation (LIC) flag values for the second merge list based on template matching cost, or b) the one or more LIC flag values for the second merge list by changing a respective value one or more LIC flags of the first merge list for the second merge list.

8 . The method of claim 1 , wherein selecting the merge list comprises:

selecting, based on the value of the flag, the second merge list for the current block.

9 . A device for decoding video data, the device comprising: one or more memories for storing the video data; and one or more processors implemented in circuitry and communicatively coupled to the one or more memories, the one or more processors configured to: determine to decode a current block of the video data using a merge mode; generate a first merge list for the current block including applying template matching to candidates of the first merge list for the current block; generate, based on the first merge list for the current block, a second merge list for the current block; obtaining a flag from a bitstream, the flag comprising a value indicative of the first merge list for the current block or the second merge list for the current block; select, based on the value of the flag, from among the first merge list for the current block and the second merge list for the current block, a merge list; and decode the current block using the merge mode and based on the selected merge list.

10 . The device of claim 9 , wherein as part of generating the second merge list, the one or more processors are configured to, prior to applying template matching to the candidates of the first merge list, change all of a plurality of local illumination compensation (LIC) flag values associated with the first merge list to opposite values for the second merge list.

11 . The device of claim 9 , wherein as part of generating the second merge list, the one or more processors are configured to, after applying template matching to the candidates of the first merge list, change all of a plurality of local illumination compensation (LIC) flag values associated with the first merge list to opposite values for the second merge list.

12 . The device of claim 9 , wherein as part of generating the second merge list, the one or more processors are configured to, prior to applying template matching to the candidates of the first merge list, trim candidates of the first merge list for the second merge list.

13 . The device of claim 9 , wherein as part of generating the second merge list, the one or more processors are configured to, after applying template matching to the candidates of the first merge list, trim candidates of the first merge list for the second merge list.

14 . The device of claim 9 , wherein as part of generating the second merge list, the one or more processors are configured to:

determine that one or more candidates of the first merge list are bi-directional prediction merge candidates; and

for each of the bi-directional prediction merge candidates, change a respective local illumination compensation (LIC) flag value associated with the first merge list to an opposite LIC flag value for the second merge list.

15 . The device of claim 9 , wherein as part of generating the second merge list, the one or more processors are configured to:

determine whether a block size of the current block is greater than a threshold; and

based on a determination that the block size of the current block is greater than the threshold, determine either a) one or more local illumination compensation (LIC) flag values for the second merge list based on template matching cost, or b) the one or more LIC flag values for the second merge list by changing a respective value one or more LIC flags of the first merge list for the second merge list.

16 . The device of claim 9 , wherein as part of selecting the merge list, the one or more processors are configured to: select, based on the value of the flag, the second merge list.

17 . Non-transitory computer-readable storage media storing instructions, which when executed, cause one or more processors to: determine to decode a current block of video data using a merge mode; generate a first merge list for the current block including applying template matching to candidates of the first merge list for the current block; generate, based on the first merge list for the current block, a second merge list for the current block; obtaining a flag from a bitstream, the flag comprising a value indicative of the first merge list for the current block or the second merge list for the current block; select, based on the value of the flag, from among the first merge list for the current block and the second merge list for the current block, a merge list; and decode the current block using the merge mode and based on the selected merge list or the second merge list.

18 . The non-transitory computer-readable storage media of claim 17 , wherein as part of generating the second merge list, the instructions cause the one or more processors to, prior to applying template matching to the candidates of the first merge list, change all of a plurality of local illumination compensation (LIC) flag values associated with the first merge list to opposite values for the second merge list.

19 . The non-transitory computer-readable storage media of claim 17 , wherein as part of generating the second merge list, the instructions cause the one or more processors to, after applying template matching to the candidates of the first merge list, change all of a plurality of local illumination compensation (LIC) flag values associated with the first merge list to opposite values for the second merge list.

20 . The non-transitory computer-readable storage media of claim 17 , wherein as part of selecting the merge list, the instructions cause the one or more processors to: and select, based on the value of the flag, the second merge list.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 30, 2024
From: ZHANG, YAN; SEREGIN, VADIM; WANG, HONGTAO; ZHANG, ZHI; CHEN, CHUN-CHI; HUANG, HAN; KARCZEWICZ, MARTA
To: QUALCOMM INCORPORATED
Reel/Frame 068741/0474 →
Continuity (3)
Provisional Application 63605138 · Dec 1, 2023
Provisional Application 63588557 · Oct 6, 2023
Related Publication 20250119565A1 · Apr 10, 2025
References Cited (51)
US 11589072B2 · Park et al. · 2023 [cited by applicant]
US 20210168403A1 · Abe · 2021 [cited by examiner]
US 20210218955A1 · Lee · 2021 [cited by examiner]
US 20210281877A1 · Liu · 2021 [cited by examiner]
US 20220078474A1 · Tamse · 2022 [cited by examiner]
US 20220116621A1 · Urban et al. · 2022 [cited by applicant]
US 20220191502A1 · Chen et al. · 2022 [cited by applicant]
US 20220239899A1 · Zhang · 2022 [cited by examiner]
US 20220329822A1 · Chang · 2022 [cited by examiner]
US 20220329823A1 · Chen · 2022 [cited by examiner]
US 20230018401A1 · Galpin et al. · 2023 [cited by applicant]
US 20230328276A1 · Zhang et al. · 2023 [cited by applicant]
US 20240022729A1 · Huang et al. · 2024 [cited by applicant]
US 20240121426A1 · Huang et al. · 2024 [cited by applicant]
US 20240195952A1 · Deng · 2024 [cited by examiner]
US 20240388691A1 · Chen et al. · 2024 [cited by applicant]
US 20250113052A1 · Li · 2025 [cited by examiner]
US 20250113062A1 · Chernyak · 2025 [cited by examiner]
US 20250126240A1 · Kim · 2025 [cited by examiner]
US 20250193389A1 · Xiu et al. · 2025 [cited by applicant]
Chen J., et al., “Non-EE2: DMVR For Affine Merge Coded Blocks”, 27. JVET Meeting, Jul. 13, 2022-Jul. 22, 2022, Teleconference; (The Joint Video Exploration Team of ISO/IEC JTC1/SC29/WG11 and ITU-T SG.16 ), No. JVET-AA01… [cited by applicant]
Coban M., et al., “Algorithm Description of Enhanced Compression Model 9 (ECM 9)”, JVET-AD2025, Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29, 30th Meeting, Antalya, TR, 2128 Apr. 2023, 142… [cited by applicant]
Hordes (Technicolor) P., et al., “CE1-5.4 Mutual Exclusion of LIC and BPWA”, 14. JVET Meeting, Mar. 19, 2019-Mar. 27, 2019, Geneva, (The Joint Video Exploration Team of ISO/IEC JTC1/SC29/WG11 and ITU-T SG.16 ), No. JVET… [cited by applicant]
Huang (Qualcomm) H., et al., “EE2-related: Sub-Block Processing for Affine DMVR”, 28. JVET Meeting, Oct. 21, 2022-Oct. 28, 2022, Mainz, (The Joint Video Exploration Team of ISO/IEC JTC1/SC29/WG11 and ITU-T SG.16 ), No. … [cited by applicant]
International Search Report and Written Opinion—PCT/US2024/046426—ISA/EPO—Dec. 9, 2024 13 Pages. [cited by applicant]
Wang Y (Bytedance)., et al., “Non-EE2: Local Illumination Compensation with Slope Adjustment”, 31. JVET Meeting, Jul. 11, 2023-Jul. 19, 2023, Geneva, CH, (The Joint Video Exploration Team of ISO/IEC JTC1/SC29/WG11 and I… [cited by applicant]
Zhang N (Bytedance Inc)., et al., “Non-EE2: LIC Flag Derivation of Merge Candidates with Template Costs”, 31. JVET Meeting, Jul. 11, 2023-Jul. 19, 2023, Geneva, (The Joint Video Exploration Team of ISO/IEC JTC1/SC29/WG1… [cited by applicant]
Zhang N (Bytedance Inc)., et al., “Non-EE2: LIC Flag Derivation of Merge Candidates with Template Costs”, JVET-AE0109-v2, Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29, 31. JVET Meeting, Ju… [cited by applicant]
Zhang Y (Qualcomm Incorporated)., et al., “Non-EE2: On LIC Flag in Merge Mode”, 32. JVET Meeting, Oct. 13, 2023-Oct. 20, 2023, Hannover, (The Joint Video Exploration Team of ISO/IEC JTC1/SC29/WG11 and ITU-T SG.16 ), No.… [cited by applicant]
Chen J., et al., “Non-EE2: DMVR for affine merge coded blocks”, JVET-AA0144-v2, Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29, 27th Meeting, by teleconference, Jul. 13-22, 2022, pp. 1-5. [cited by applicant]
Chen Y., et al., “Description of SDR, HDR and 360° Video Coding Technology Proposal by Qualcomm and Technicolor—Low and High Complexity Versions”, 10th JVET Meeting, Apr. 10, 2018-Apr. 20, 2018, San Diego, (The Joint Vi… [cited by applicant]
Coban M., et al., “Algorithm Description of Enhanced Compression Model 10 (ECM 10)”, JVET-AE2025, Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29, 31st Meeting, Geneva, CH, Jul. 11-19, 2023, … [cited by applicant]
Coban M., et al., “Algorithm Description of Enhanced Compression Model 11 (ECM 11)”, JVET-AF2025, Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29 32nd Meeting, Hannover, DE, Oct. 13-20, 2023,… [cited by applicant]
Coban M., et al., “Algorithm Description of Enhanced Compression Model 6 (ECM 6)”, JVET-AA2025, Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29 27th Meeting, by Teleconference, Jul. 13-22, 20… [cited by applicant]
Co-pending U.S. Appl. No. 18/882,225, filed Sep. 11, 2024. [cited by applicant]
Co-pending U.S. Appl. No. 18/882,291, filed Sep. 11, 2024. [cited by applicant]
Han Y., et al., “CE4.4.6: Improvement on Merge/Skip Mode,” JVET-L0399_r2, Joint Video Exploration Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, JVET-L0399, 12th Meeting: Macao, CN, Oct. 3-12, 2018, pp. … [cited by applicant]
ITU-T H.265: “Series H: Audiovisual and Multimedia Systems Infrastructure of Audiovisual Services—Coding of Moving Video”, High Efficiency Video Coding, The International Telecommunication Union, Jun. 2019, 696 Pages. [cited by applicant]
ITU-T H.266: “Series H: Audiovisual and Multimedia Systems Infrastructure of Audiovisual Services—Coding of Moving Video”, Versatile Video Coding, The International Telecommunication Union, Aug. 2020, 516 pages. [cited by applicant]
Karczewicz M., et al., “Common Test Conditions and Evaluation Procedures for Enhanced Compression Tool Testing”, JVET-Y2017-v1, Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29, 25th Meeting, … [cited by applicant]
Salehifar M., et al., “EE2-3.5: Iterative BDOF pass in multi-pass DMVR”, JVET-AE0065, Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29, 31st Meeting, Geneva, CH, Jul. 11-19, 2023, pp. 1-3. [cited by applicant]
Seregin V., et al., “CE4-3.1a and CE4-3.1b: Unidirectional Local Illumination Compensation with Affine Prediction”, JVET-O0066-v1, Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 15th … [cited by applicant]
Seregin V., “Merge Branch ‘fix-keep-pred-and-resi-signals’ into ‘master’,” ECM software, Created on Jul. 22, 2021, pp. 1-3. [cited by applicant]
Xiu X., et al., “EE2-Test2.7: Improvements on Local Illumination Compensation”, JVET-AD0213-v1, Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29 30th Meeting, Antalya, TR, Apr. 21-28, 2023, pp… [cited by applicant]
Xiu X., et al., “Non-EE2: Enhancements on local illumination compensation”, JVET-AF0191-v2, Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29, 32nd Meeting, Hannover, DE, Oct. 13-20, 2023, pp. … [cited by applicant]
Zhang N., et al., “EE2-3.2: LIC flag derivation for merge candidates with template costs”, JVET-AF0128-v2, Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29, 32nd Meeting, Hannover, DE, Oct. 13… [cited by applicant]
Zhang Y., et al., “EE2: Test 2.1, 2.2, 2.5, 2.6a, 2.6e and 2.6f on LIC improvement”, JVET-AG0176-v2, Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29, 33rd Meeting, by teleconference, Jan. 17-… [cited by applicant]
Zhang Y., et al., “EE2-2.1: Regression Based Affine Candidate Derivation”, JVET-AA0107-v2, Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29, 27th Meeting, by teleconference, Jul. 13-22, 2022, … [cited by applicant]
Zhang Y., et al., “Non-EE2: On LIC flag in merge mode”, JVET-AF0194, Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29, 32nd Meeting, Hannover, DE, Oct. 13-20, 2023, pp. 1-2. [cited by applicant]
Bordes P., et al., “CE1-5.4: Mutual Exclusion of LIC and BPWA”, JVET-N0283-v2, Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC29/WG 11 14th Meeting: Geneva, CH, Mar. 19-27, 2019, 3 Pages. [cited by applicant]
Huang H., et al., “EE2-Related: Sub-block Processing for Affine DMVR”, JVET-AB0177-v2, Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29 28th Meeting, Mainz, DE, Oct. 20-28, 2022, 3 Pages. [cited by applicant]