IP Library › Granted Patent US 12,593,038
Granted Patent B2
US 12,593,038 · App. 18/449,134 · Granted Mar 31, 2026

Temporal prediction of parameters in non-linear adaptive loop filter

Inventors: Li Zhang (San Diego, CA); Kai Zhang (San Diego, CA); Hongbin Liu (Beijing, CN); Yue Wang (Beijing, CN)
Assignees: BEIJING BYTEDANCE NETWORK TECHNOLOGY CO., LTD.; BYTEDANCE INC.
H04N19/117H04N19/119H04N19/132H04N19/174H04N19/176H04N19/184H04N19/70H04N19/82
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,593,038
App. No.
18/449,134
Granted
Mar 31, 2026
Kind
B2
Abstract

Devices, systems and methods for temporal prediction of parameters in non-linear adaptive loop filtering are described. In an exemplary aspect, a method for visual media processing includes configuring, for a current video block, one or more parameters of a clipping operation that is part of a non-linear filtering operation; and performing, based on the one or more parameters, a conversion between the current video block and a bitstream representation of the current video block, wherein the one or more parameters is coded in accordance with a rule.

Claims (83)

1 . A method of processing video data, comprising:

determining, that a non-linear filtering operation is applied for a current video region of a video;

determining, for the current video region, one or more parameters of a clipping operation that is part of the non-linear filtering operation; and

performing, based on the one or more parameters, a conversion between the current video region and a bitstream of the video,

wherein at least one first filtering index is derived, and a first clipping parameter set is derived based on the at least one first filtering index and the one or more parameters,

wherein the one or more parameters are coded with a fixed length of N bits,

wherein the at least one first filtering index is derived based on multiple sample differences in different directions,

wherein the current video region is split into multiple M×M video sub-regions, and the multiple sample differences in different directions are derived for every M×M video sub-region,

wherein M is equal to 2 or 4, and

wherein the multiple sample differences in different directions are derived based on a 1:P subsampling rate, and wherein P is greater than 1.

2 . The method of claim 1 , wherein N is 2.

3 . The method of claim 1 , wherein the one or more parameters include a clipping index syntax element.

4 . The method of claim 3 , wherein a first filtering coefficient set is derived based on the at least one first filtering index and a coefficient parameter syntax element.

5 . The method of claim 4 , wherein a value of the clipping index syntax element is determined independently from the first filtering coefficient set, and

wherein in a case that values of the first filtering coefficient set are zero, the clipping index syntax element is included in the bitstream.

6 . The method of claim 1 , wherein the one or more parameters are included as fields in an adaptive parameter set.

7 . The method of claim 5 , wherein the clipping index syntax element and the coefficient parameter syntax element are present in a same adaptive parameter set.

8 . The method of claim 4 , wherein the coefficient parameter syntax element includes a coefficient absolute value.

9 . The method of claim 4 , wherein the coefficient parameter syntax element is coded using Exponential-Golomb model with fixed order.

10 . The method of claim 1 , wherein the current video region includes a coding tree block or a slice.

11 . The method of claim 1 , wherein the conversion includes encoding the current video region into the bitstream.

12 . The method of claim 1 , wherein the conversion includes decoding the current video region from the bitstream.

13 . An apparatus for processing video data comprising a processor and a non-transitory memory with instructions thereon, wherein the instructions upon execution by the processor, cause the processor to:

determine, that a non-linear filtering operation is applied for a current video region of a video;

determine, for the current video region, one or more parameters of a clipping operation that is part of the non-linear filtering operation; and

perform, based on the one or more parameters, a conversion between the current video region and a bitstream of the video,

wherein at least one first filtering index is derived, and a first clipping parameter set is derived based on the at least one first filtering index and the one or more parameters, and

wherein the one or more parameters are coded with a fixed length of N bits,

wherein the at least one first filtering index is derived based on multiple sample differences in different directions,

wherein the current video region is split into multiple M×M video sub-regions, and the multiple sample differences in different directions are derived for every M×M video sub-region,

wherein M is equal to 2 or 4, and

wherein the multiple sample differences in different directions are derived based on a 1:P subsampling rate, and wherein P is greater than 1.

14 . The apparatus of claim 13 , wherein N is 2,

wherein the one or more parameters include a clipping index syntax element,

wherein a first filtering coefficient set is derived based on the at least one first filtering index and a coefficient parameter syntax element,

wherein a value of the clipping index syntax element is determined independently from the first filtering coefficient set,

wherein in a case that values of the first filtering coefficient set are zero, the clipping index syntax element is included in the bitstream,

wherein the one or more parameters are included as fields in an adaptive parameter set,

wherein the clipping index syntax element and the coefficient parameter syntax element are present in a same adaptive parameter set,

wherein the coefficient parameter syntax element includes a coefficient absolute value,

wherein the coefficient parameter syntax element is coded using Exponential-Golomb model with fixed order, and

wherein the current video region includes a coding tree block or a slice.

15 . A non-transitory computer-readable storage medium storing instructions that cause a processor to:

determine, that a non-linear filtering operation is applied for a current video region of a video;

determine, for the current video region, one or more parameters of a clipping operation that is part of the non-linear filtering operation; and

perform, based on the one or more parameters, a conversion between the current video region and a bitstream of the video,

wherein at least one first filtering index is derived, and a first clipping parameter set is derived based on the at least one first filtering index and the one or more parameters, and

wherein the one or more parameters are coded with a fixed length of N bits,

wherein the at least one first filtering index is derived based on multiple sample differences in different directions,

wherein the current video region is split into multiple M×M video sub-regions, and the multiple sample differences in different directions are derived for every M×M video sub-region,

wherein M is equal to 2 or 4, and

wherein the multiple sample differences in different directions are derived based on a 1:P subsampling rate, and wherein P is greater than 1.

16 . The non-transitory computer-readable storage medium of claim 15 , wherein N is 2,

wherein the one or more parameters include a clipping index syntax element,

wherein a first filtering coefficient set is derived based on the at least one first filtering index and a coefficient parameter syntax element,

wherein a value of the clipping index syntax element is determined independently from the first filtering coefficient set,

wherein in a case that values of the first filtering coefficient set are zero, the clipping index syntax element is included in the bitstream,

wherein the one or more parameters are included as fields in an adaptive parameter set,

wherein the clipping index syntax element and the coefficient parameter syntax element are present in a same adaptive parameter set,

wherein the coefficient parameter syntax element includes a coefficient absolute value,

wherein the coefficient parameter syntax element is coded using Exponential-Golomb model with fixed order, and

wherein the current video region includes a coding tree block or a slice.

17 . A method for storing a bitstream of a video, comprising:

determining, that a non-linear filtering operation is applied for a current video region of the video;

determining, for the current video region, one or more parameters of a clipping operation that is part of the non-linear filtering operation;

generating, based on the one or more parameters, for the current video region, the bitstream of the video, and

storing the bitstream in a non-transitory computer-readable recording medium,

wherein at least one first filtering index is derived, and a first clipping parameter set is derived based on the at least one first filtering index and the one or more parameters, and

wherein the one or more parameters are coded with a fixed length of N bits,

wherein the at least one first filtering index is derived based on multiple sample differences in different directions,

wherein the current video region is split into multiple M×M video sub-regions, and the multiple sample differences in different directions are derived for every M×M video sub-region,

wherein M is equal to 2 or 4, and

wherein the multiple sample differences in different directions are derived based on a 1:P subsampling rate, and wherein P is greater than 1.

18 . The method of claim 17 , wherein N is 2,

wherein the one or more parameters include a clipping index syntax element,

wherein a first filtering coefficient set is derived based on the at least one first filtering index and a coefficient parameter syntax element,

wherein a value of the clipping index syntax element is determined independently from the first filtering coefficient set,

wherein in a case that values of the first filtering coefficient set are zero, the clipping index syntax element is included in the bitstream,

wherein the one or more parameters are included as fields in an adaptive parameter set,

wherein the clipping index syntax element and the coefficient parameter syntax element are present in a same adaptive parameter set,

wherein the coefficient parameter syntax element includes a coefficient absolute value,

wherein the coefficient parameter syntax element is coded using Exponential-Golomb model with fixed order, and

wherein the current video region includes a coding tree block or a slice.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 21, 2023
From: ZHANG, LI; ZHANG, KAI
To: BYTEDANCE INC.
Reel/Frame 064652/0117 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 21, 2023
From: LIU, HONGBIN; WANG, YUE
To: BEIJING BYTEDANCE NETWORK TECHNOLOGY CO., LTD.
Reel/Frame 064652/0161 →
Priority Claims (1)
WO PCT/CN2019/082626 · Apr 15, 2019 · international
Continuity (4)
Continuation 17646048 · Dec 27, 2021
Continuation 17392628 · Aug 3, 2021
Continuation PCTCN2020084876 · Apr 15, 2020
Related Publication 20230388492A1 · Nov 30, 2023
References Cited (113)
US 8929462B2 · Wang · 2015 [cited by applicant]
US 9591331B2 · Chong · 2017 [cited by applicant]
US 9813719B2 · Wang · 2017 [cited by applicant]
US 10057574B2 · Li · 2018 [cited by applicant]
US 10271052B2 · Zhang · 2019 [cited by applicant]
US 10277909B2 · Ye · 2019 [cited by applicant]
US 10404988B2 · Ye · 2019 [cited by applicant]
US 10506230B2 · Zhang · 2019 [cited by applicant]
US 10582213B2 · Li · 2020 [cited by applicant]
US 10694202B2 · Zhang · 2020 [cited by applicant]
US 10694204B2 · Chen · 2020 [cited by applicant]
US 10721469B2 · Zhang · 2020 [cited by applicant]
US 10764576B2 · Li · 2020 [cited by applicant]
US 10778974B2 · Karczewicz · 2020 [cited by applicant]
US 10855985B2 · Zhang · 2020 [cited by applicant]
US 11233995B2 · Zhang · 2022 [cited by applicant]
US 11303894B2 · Zhang · 2022 [cited by applicant]
US 20010051003A1 · Henderson · 2001 [cited by examiner]
US 20070058713A1 · Shen · 2007 [cited by applicant]
US 20080239090A1 · Ohwaki · 2008 [cited by applicant]
US 20090116558A1 · Chen · 2009 [cited by applicant]
US 20110142136A1 · Liu · 2011 [cited by applicant]
US 20130107973A1 · Wang · 2013 [cited by applicant]
US 20130140801A1 · Schlee · 2013 [cited by applicant]
US 20130258049A1 · Chong · 2013 [cited by applicant]
US 20150049819A1 · Lee · 2015 [cited by examiner]
US 20150049821A1 · Chen · 2015 [cited by applicant]
US 20160212002A1 · Xiong · 2016 [cited by applicant]
US 20160323581A1 · Zhang · 2016 [cited by examiner]
US 20160366422A1 · Yin · 2016 [cited by applicant]
US 20170238020A1 · Karczewicz · 2017 [cited by applicant]
US 20170332085A1 · Ramasubramonian · 2017 [cited by examiner]
US 20180041778A1 · Zhang · 2018 [cited by applicant]
US 20180041779A1 · Zhang · 2018 [cited by applicant]
US 20190238845A1 · Zhang · 2019 [cited by applicant]
US 20190306502A1 · Gadde · 2019 [cited by applicant]
US 20190394463A1 · Chen · 2019 [cited by examiner]
US 20200177910A1 · Li · 2020 [cited by applicant]
US 20200267381A1 · Vanam · 2020 [cited by applicant]
US 20200304785A1 · Hu · 2020 [cited by applicant]
US 20200314423A1 · Hu · 2020 [cited by examiner]
US 20200322632A1 · Hanhart · 2020 [cited by applicant]
US 20200359016A1 · Li · 2020 [cited by applicant]
US 20200359017A1 · Li · 2020 [cited by applicant]
US 20200359018A1 · Li · 2020 [cited by applicant]
US 20200413038A1 · Zhang · 2020 [cited by applicant]
US 20210044834A1 · Li · 2021 [cited by applicant]
US 20210084295A1 · Chen · 2021 [cited by applicant]
US 20210084318A1 · Kuo · 2021 [cited by applicant]
US 20210211657A1 · Ikonin · 2021 [cited by applicant]
US 20210274170A1 · Du · 2021 [cited by applicant]
US 20210368171A1 · Zhang · 2021 [cited by applicant]
US 20210377524A1 · Zhang · 2021 [cited by applicant]
CN 102055981A · 2011 [cited by applicant]
CN 106031172A · 2016 [cited by applicant]
CN 107211154A · 2017 [cited by applicant]
CN 107909618A · 2018 [cited by applicant]
CN 108028920A · 2018 [cited by applicant]
CN 108141602A · 2018 [cited by applicant]
CN 108605143A · 2018 [cited by applicant]
CN 109076218A · 2018 [cited by applicant]
CN 109417639A · 2019 [cited by applicant]
EP 3244611A1 · 2017 [cited by applicant]
EP 3342163A1 · 2018 [cited by applicant]
JP 2014520451A · 2014 [cited by applicant]
JP 2020017970A · 2020 [cited by applicant]
JP 7405865B2 · 2023 [cited by applicant]
TW 201904285A · 2019 [cited by applicant]
WO 2011125866A1 · 2011 [cited by applicant]
WO 2014007735A1 · 2014 [cited by applicant]
WO 2017045101A1 · 2017 [cited by applicant]
WO 2017045580A1 · 2017 [cited by applicant]
WO 2017194312A2 · 2017 [cited by applicant]
WO 2018123800A1 · 2018 [cited by applicant]
WO 2018135830A1 · 2018 [cited by applicant]
WO 2019065487A1 · 2019 [cited by applicant]
WO 2020182620A1 · 2020 [cited by applicant]
WO 2020200277A1 · 2020 [cited by applicant]
Taquet et al, “Non-Linear Adaptive Loop Filter” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 13th Meeting: Marrakech, MA, 9â18 Jan. 2019. [cited by examiner]
Taquet et al. “CE5: Results of tests CE5-3.1 to CE5-3.4 on Non-Linear Adaptive Loop Filter.” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 14th Meeting: Geneva, CH, 19â27 Mar. 2019. [cited by examiner]
Lim et al. “CE2: Subsampled Laplacian Calculation (Test 6.1, 6.2, 6.3, and 6.4),” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 12th Meeting, Macao, CN, Oct. 3-12, 2018, document JVE… [cited by applicant]
Taquet et al. “CE5: Results ofTests CE5-3.1 to CE5-3.4 on Non-Linear Adaptive Filter,” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 14th Meeting, Geneva, CH, Mar. 19-27, 2019, docum… [cited by applicant]
Zhao et al. “CE3-Related: Harmonization Between ISP and WAIP,” Joint Video Experts Team {JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 14th Meeting, Geneva, CH, Mar. 19-27, 2019, document JVET-N0427, 2019. [cited by applicant]
Rasch et al. “CE10: Uniform Directional Diffusion Filters for Video Coding,” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 12th Meeting, Macao, CN, Oct. 3-12, 2018, document JVET-L01… [cited by applicant]
Rusanovskyy et al. “CE14: Test on In-Loop Bilateral Filter From JVET-J0021/JVET-K0384 with Parametrization (CE14.2),” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 12th Meeting, Maca… [cited by applicant]
Stepin et al. “CE2 Related: Hadamard Transform Domain Filter,” Joint Video Experts Team (JVET) of ITU-T SG 16 and ISO/IEC JTC 1/SC 29/WG 11, 11th Meeting, Ljubljana, SI, Jul. 10-18, 2018, document JVET-K0068, 2018. [cited by applicant]
“Information Technology—High Efficiency Coding and Media Delivery in Heterogeneous Environments—Part 2: High Efficiency Video Coding” Apr. 20, 2018, ISO/DIS 23008, 4th Edition. [cited by applicant]
Chen et al. “Algorithm Description of Joint Exploration Test Model 7 (JEM 7),” Joint Video Exploration Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 7th Meeting: Torino, IT, Jul. 13-21, 2017, document J… [cited by applicant]
Karczewicz et al. “Geometry Tranformation-based Adaptive In-Loop Filter,” 2016, IEEE Qualcomm Technologies Inc. San Diego, CA, USA. [cited by applicant]
Taquet et al. “CE5: Complementary Results ofTests CE5-3 on Non-Linear ALF,” Joint Video Experts Team {JVET) of ITU-T SG 16 WP 3 and 1SO/IEC JTC 1/SC 29/WG 11, 14th Meeting, Geneva, CH, Mar. 19-27, 2019, document JVET-N0… [cited by applicant]
Taquet et al. “Non-Linear Adaptive Loop Filter,” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO-IEC JTC 1/SC 29/WG 11, 13th Meeting, Marrakech, MA, Jan. 9-8, 2019, documentJVET-M0385, 2019. [cited by applicant]
Lin et al. “CE13: HEC with In-Loop Filters Using Spherical Neighbors (Test 1.3),” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 13th Meeting, Marrakech, MA, Jan. 9-18, 2019, document… [cited by applicant]
Chen et al. “CE5-1: Adaptive Loop Filter with Virtual Boundary Processing,” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and 1SO/IEC JTC 1/SC 29/WG 11, 14th Meeting, Geneva, CH, Mar. 19-27, 2019, document JVET-N0… [cited by applicant]
JVET-M1001—Bross et al. “Versatile Video Coding (Draft 4),” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 13th Meeting: Marrakech, MA, Jan. 9-18, 2019, document JVET-M1001, 2019. [cited by applicant]
Chen et al. “Algorithm Description of Joint Exploration Test Model 4,” Joint Video Exploration Team (JVET) of ITU-T BG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 4th Meeting: Chengdu, CN, Oct. 15-21, 2016, documentJVET-D100… [cited by applicant]
JCTVC-F455-v3-T. Nguyen, et al., “Modified binarization and coding of MVD for PIPE/CABAC”, Document: JCTVC-F455, [online], Joint Collaborative Team on Video Coding (JCT-VC) of ITU-T SG16 WP3 and ISO/IEC JTC1/SC29/ WG11,… [cited by applicant]
Boyce, Jill. “Update to Interoperability Point Syntax,” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and SO/IEC JTC 1/SC 29/WG 11 13th Meeting: Marrakech, MA, Jan. 9-18, 2019, document JVET-M0451, 2019, p. 27-29 … [cited by applicant]
Bross et al. “Versatile Video Coding (Draft 3),” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 12th Meeting: Macao, CN, Oct. 3-12, 2018, document JVET-L1001, 2018. [cited by applicant]
Boyce, Jill, “Proposed Starting Point for Interoperability Point Syntax,” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 12th Meeting: Macao, CN, Oct. 3-12, 2018, document JVET-L0696, … [cited by applicant]
JVET-J0013-v2-Jung Won Kang, et al., “Description of SDR video coding technology proposal by ETRI and Sejong University”, Document: JVET-J0013-v2, [online], Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IE… [cited by applicant]
International Search Report and Written Opinion from International Patent Application No. PCT/CN2020/084873 dated Jun. 29, 2020 {10 pages). [cited by applicant]
International Search Report and Written Opinion from International Patent Application No. PCT/CN2020/084876 dated Jun. 30, 2020 {10 pages). [cited by applicant]
Non Final Office Action from U.S. Appl. No. 17/392,581 dated Oct. 26, 2021. [cited by applicant]
Notice of Allowance from U.S. Appl. No. 17/392,628 dated Nov. 10, 2021. [cited by applicant]
Extended European Search Report from European Patent Application No. 20791044.9 dated May 19, 2022 (10 pages). [cited by applicant]
Examination Report from Indian Patent Application No. 202127043318 dated May 5, 2022 (6 pages). [cited by applicant]
Non Final Office Action from U.S. Appl. No. 17/646,048 dated Nov. 18, 2022 (34 pages). [cited by applicant]
Bross B., et al., “Versatile Video Coding (Draft 4),” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 13th Meeting: Marrakech, MA, Jan. 9-18, 2019, Document: JVET-M1001-v5, 287 pages. [cited by applicant]
Document: JVET-M0385, Taquet J., “Non-linear Adaptive Loop Filter,” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 13th Meeting: Marrakech, MA, Jan. 9-18, 2019, 3 pages. [cited by applicant]
Document: JVET-O0290-v2, Chubach, O., et al., “CE5-related: Modified syntax design of ALF clipping parameters signalling,” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 15th Meeting: … [cited by applicant]
Document: JVET-O0290-v4, Chubach, O., et al., “CE5-related: Modified syntax design of ALF clipping parameters signalling,” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 15th Meeting: … [cited by applicant]
Chinese Office Action from Chinese Patent Application No. 202211656192.2 dated Jan. 20, 2026, 45 pages. [cited by applicant]
Japanese Office Action from Japanese Patent Application No. 2023-201865 dated Feb. 10, 2026, 8 pages. [cited by applicant]