IP Library Granted Patent US 12,348,787
Granted Patent B2
US 12,348,787 · App. 18/140,806 · Granted Jul 1, 2025

Methods for simplifying adaptive loop filter in video coding

Inventors: Rahul Vanam (San Diego, CA); Yuwen He (San Diego, CA); Yan Ye (San Diego, CA)
Assignee: InterDigital VC Holdings, Inc.
H04N19/82G06F18/211G06F18/241G06F18/2431H04N19/105H04N19/117H04N19/176
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,348,787
App. No.
18/140,806
Granted
Jul 1, 2025
Kind
B2
Abstract

Systems, methods and instrumentalities are disclosed for adaptively selecting an adaptive loop filter (ALF) procedure for a frame based on which temporal layer the frame is in. ALF procedures may vary in computational complexity. One or more frames including the current frame may be in a temporal layer of a coding scheme. The decoder may determine the current frame's temporal layer level within the coding scheme. The decoder may select an ALF procedure based on the current frame's temporal layer level. If the current frame's temporal layer level is higher within the coding scheme than some other temporal layer levels, an ALF procedure that is less computationally complex may be selected for the current frame. Then the decoder may perform the selected ALF procedure on the current frame.

Claims (58)

1. A video decoding device, comprising:

a processor configured to:

select, from a plurality of pixels, a subset of pixels of a video block;

determine a first diagonal variation using a first pixel of the subset of pixels and a second pixel of the subset of pixels, wherein the second pixel is positioned in a first diagonal direction from the first pixel;

obtain a sum of diagonal variations, wherein the sum of diagonal variations is a first sum of diagonal variations and is obtained based on the first diagonal variation;

determine a diagonal gradient based on the sum of diagonal variations;

determine a second diagonal variation using a third pixel of the subset of pixels and a fourth pixel of the subset of pixels, wherein the fourth pixel is positioned in a second diagonal direction from the third pixel;

obtain a second sum of diagonal variations using the second diagonal variation;

perform adaptive loop filtering (ALF) on the video block based on the diagonal gradient and the second sum of diagonal variations; and

decode a picture comprising the video block.

2. The video decoding device of claim 1 , wherein, for the selection of the subset of pixels of the video block, a pixel of the plurality of pixels is skipped in a diagonal direction.

3. The video decoding device of claim 1 , wherein the performance of ALF on the video block comprises classification of the video block, and wherein the classification of the video block is based on the diagonal gradient.

4. The video decoding device of claim 1 , wherein the diagonal gradient is a first diagonal gradient associated with a first diagonal direction, and the processor is further configured to determine a second diagonal gradient associated with a second diagonal direction, and wherein the ALF is performed on the video block further based on the second diagonal gradient.

5. The video decoding device of claim 1 , wherein the first pixel and the third pixel are the same, and the second pixel and the fourth pixel are different.

6. The video decoding device of claim 1 , wherein, for the selection of the subset of pixels, a pixel is skipped in a vertical direction, and a pixel is skipped in a horizontal direction.

7. The video decoding device of claim 1 , wherein the processor is further configured to determine, using the subset of pixels, a horizontal gradient and a vertical gradient, wherein the ALF is performed further based on the horizontal gradient and the vertical gradient.

8. A video decoding method, comprising:

selecting, from a plurality of pixels, a subset of pixels of a video block;

determining a first diagonal variation using a first pixel of the subset of pixels and a second pixel of the subset of pixels, wherein the second pixel is positioned in a first diagonal direction from the first pixel;

obtaining a sum of diagonal variations, wherein the sum of diagonal variations is a first sum of diagonal variations and is obtained based on the first diagonal variation;

determining a diagonal gradient based on the sum of diagonal variations;

determining a second diagonal variation using a third pixel of the subset of pixels and a fourth pixel of the subset of pixels, wherein the fourth pixel is positioned in a second diagonal direction from the third pixel;

obtaining a second sum of diagonal variations using the second diagonal variation;

performing adaptive loop filtering (ALF) on the video block based on the diagonal gradient and the second sum of diagonal variations; and

decoding a picture comprising the video block.

9. The video decoding method of claim 8 , wherein the selection of the subset of pixels of the video block comprises skipping a pixel of the plurality of pixels in a diagonal direction.

10. The video decoding method of claim 8 , wherein the performance of ALF on the video block comprises classification of the video block, and wherein the classification of the video block is based on the diagonal gradient.

11. A video encoding device, comprising:

a processor configured to:

select, from a plurality of pixels, a subset of pixels of a video block;

determine a first diagonal variation using a first pixel of the subset of pixels and a second pixel of the subset of pixels, wherein the second pixel is positioned in a first diagonal direction from the first pixel;

obtain a sum of diagonal variations, wherein the sum of diagonal variations is a first sum of diagonal variations and is obtained based on the first diagonal variation;

determine a diagonal gradient based on the sum of diagonal variations;

determine a second diagonal variation using a third pixel of the subset of pixels and a fourth pixel of the subset of pixels, wherein the fourth pixel is positioned in a second diagonal direction from the third pixel;

obtain a second sum of diagonal variations using the second diagonal variation;

perform adaptive loop filtering (ALF) on the video block based on the diagonal gradient and the second sum of diagonal variations; and

encode a picture comprising the video block.

12. The video encoding device of claim 11 , wherein the processor is further configured to:

determine a first error associated with the picture;

determine a second error based on ALF associated with the picture;

compare the first error and the second error using a bias factor; and

include, based on the comparison, an ALF indication in video data to indicate an ALF utilization status associated with the picture.

13. The video encoding device of claim 11 , wherein, for the selection of the subset of pixels of the video block, a pixel of the plurality of pixels is skipped in a diagonal direction.

14. A video encoding method, comprising:

selecting, from a plurality of pixels, a subset of pixels of a video block;

determining a first diagonal variation using a first pixel of the subset of pixels and a second pixel of the subset of pixels, wherein the second pixel is positioned in a first diagonal direction from the first pixel;

obtaining a sum of diagonal variations, wherein the sum of diagonal variations is a first sum of diagonal variations and is obtained based on the first diagonal variation;

determining a diagonal gradient based on the sum of diagonal variations;

determining a second diagonal variation using a third pixel of the subset of pixels and a fourth pixel of the subset of pixels, wherein the fourth pixel is positioned in a second diagonal direction from the third pixel;

obtaining a second sum of diagonal variations using the second diagonal variation;

performing adaptive loop filtering (ALF) on the video block based on the diagonal gradient and the second sum of diagonal variations; and

encoding a picture comprising the video block.

15. The video encoding method of claim 14 , wherein the method further comprises:

determining a first error associated with the picture;

determining a second error based on ALF associated with the picture;

comparing the first error and the second error using a bias factor; and

including, based on the comparison, an ALF indication in video data to indicate an ALF utilization status associated with the picture.

16. The video encoding method of claim 14 , wherein the diagonal gradient is a first diagonal gradient associated with a first diagonal direction, and the method further comprises determining a second diagonal gradient associated with a second diagonal direction, and wherein the ALF is performed on the video block further based on the second diagonal gradient.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 12, 2024
From: VANAM, RAHUL; HE, YUWEN; YE, YAN
To: VID SCALE, INC.
Reel/Frame 068572/0114 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jul 11, 2024
From: VID SCALE, INC.
To: INTERDIGITAL VC HOLDINGS, INC.
Reel/Frame 068284/0031 →
Continuity (4)
Continuation 16760974
Provisional Application 62607033 · Dec 18, 2017
Provisional Application 62579977 · Nov 1, 2017
Related Publication 20230388553A1 · Nov 30, 2023
References Cited (56)
US 9998737B2 · Fu et al. · 2018 [cited by applicant]
US 10382766B2 · Karczewicz et al. · 2019 [cited by applicant]
US 10848760B2 · Drugeon et al. · 2020 [cited by applicant]
US 11290715B2 · Lim et al. · 2022 [cited by applicant]
US 20120039389A1 · Sjöberg et al. · 2012 [cited by applicant]
US 20120183078A1 · Lai et al. · 2012 [cited by applicant]
US 20150117555A1 · Chong et al. · 2015 [cited by applicant]
US 20150264406A1 · Kim et al. · 2015 [cited by applicant]
US 20170332075A1 · Karczewicz et al. · 2017 [cited by applicant]
US 20180041779A1 · Zhang et al. · 2018 [cited by applicant]
US 20180077414A1 · Reddy et al. · 2018 [cited by applicant]
US 20180192050A1 · Zhang et al. · 2018 [cited by applicant]
US 20190014315A1 · Karczewicz · 2019 [cited by examiner]
US 20200145651A1 · Abe et al. · 2020 [cited by applicant]
US 20200236355A1 · Zhao et al. · 2020 [cited by applicant]
US 20200304827A1 · Abe et al. · 2020 [cited by applicant]
US 20210218962A1 · Lim et al. · 2021 [cited by applicant]
US 20210314565A1 · Ohkawa et al. · 2021 [cited by applicant]
CN 102804776A · 2012 [cited by applicant]
EP 3471413A1 · 2019 [cited by applicant]
JP 2014511615A · 2014 [cited by applicant]
JP 7036628B2 · 2022 [cited by applicant]
KR 1020130053645A · 2013 [cited by applicant]
WO 2016130801A1 · 2016 [cited by applicant]
WO 2017123487A1 · 2017 [cited by applicant]
WO 2017201011A1 · 2017 [cited by applicant]
WO 2019010217A1 · 2019 [cited by applicant]
(P. Lai and F. C. A. Fernandes, “Computationally efficient adaptive loop filtering design with directional features for video coding,” 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASS… [cited by examiner]
(Li, Xiang, Yongjo Ahn, and Donggyu Sim. “Complexity Reduction of an Adaptive Loop Filter Based on Locall Homogeneity.” IEIE Transactions on Smart Processing and Computing 6.2 (2017): 93-101) (hereinafter Li) (Year: 201… [cited by examiner]
(M. Karczewicz, L. Zhang, W. Chien and X. Li, “Geometry transformation-based adaptive in-loop filter,” 2016 Picture Coding Symposium (PCS). Nuremberg, 2016, pp. 1-5, doi: 10.1109/PCS.2016.7906346) (Year: 2016). [cited by examiner]
“JEM-7.0 Reference Software”, Available at <https://jvet.hhi.fraunhofer.de/svn/svn_HMJEMSoftware/tags/HM-16.6-JEM-7.0>, 1 page. [cited by applicant]
An et al., “Unified Adaptive Loop Filter for Luma and Chroma”, JVET-G0095, HiSilicon Technologies Co., Ltd., Joint Video Exploration Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 7th Meeting: Torino, IT… [cited by applicant]
Boyce et al., “Draft High Efficiency Video Coding (HEVC) Version 2, Combined Format Range Extensions (RExt), Scalability (SHVC), and Multi-View (MV-HEVC) Extensions”, JCTVC-R1013_V1, Editors, Joint Collaborative Team on… [cited by applicant]
Chen et al., “Algorithm Description of Joint Exploration Test Model 7 (JEM 7)”, Editors, JVET-G1001-V1, Joint Video Exploration Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 7th Meeting: Torino, IT, Jul… [cited by applicant]
Chen et al., “Further Improvements to HMKTA-1.0”, VCEG-AZ07, Qualcomm Incorporated, ITU—Telecommunications Standardization Sector, Study Group 16 Question 6, Video Coding Experts Group (VCEG), 52nd Meeting, Warsaw, Pola… [cited by applicant]
Hellman, Tim, “ALF Complexity Analysis”, JCTVC-F342_r2, Torino, Italy, Jul. 2011. [cited by applicant]
ISO/IEC, “Information Technology-Coding of Moving Pictures and Associated Audio for Digital Storage Media at up to About 1,5 Mbit/s-Part 2: Video”, ISO/IEC 11172-2:1993/Cor.3:2003(E), Nov. 2003, 6 pages. [cited by applicant]
ISO/IEC, “Information Technology-Generic Coding of Moving Pictures and Associated Audio Information: Video”, ISO/IEC 13818-2:2000(E), Dec. 15, 2000, 220 pages. [cited by applicant]
ITU, “Codec For Audiovisual Services AT n*384 kbits”, Series H: Audiovisual And Multimedia Systems, Coding of Moving Video, International Telecommunication Union, ITU-T Rec H.261, Nov. 1988, 14 pages. [cited by applicant]
ITU-T, “Advanced Video Coding for Generic Audiovisual Services”, H.264, Series H: Audiovisual and Multimedia Systems, Infrastructure of Audiovisual Services—Coding of Moving Video, Nov. 2007, 564 pages. [cited by applicant]
ITU-T, “Video Coding for Low Bit Rate Communication”, ITU-T Recommendation H.263, Transmission of Non-Telephone Signals, Mar. 1996, 52 pages. [cited by applicant]
Karczewicz, Marta et al., “Geometry Transformation-based Adaptive In-Loop Filter”, IEEE; 2016 Picture Coding Symposium (PCS), Dec. 7, 2016, 5 pages. [cited by applicant]
Kim et al., “Fast Intra Mode Decision Algorithm Using Sub-Sampled Pixels”, IEEE 15th International Symposium on Consumer Electronics, 2011, pp. 290-293. [cited by applicant]
Lai et al., “Adaptive Loop Filter with Directional Features and Similarity Mapping for Video Coding”, Visual Information Processing and Communication III, vol. 8305, Feb. 2012, pp. 34-41. [cited by applicant]
Lai et al., “CE8 Subtest 1: Block-Based Filter Adaptation with Features on Subset of Pixels”, JCTVC-F301, Samsung Electronics Co., Ltd., Joint Collaborative Team on Video Coding (JCT-VC) of ITU-T SG16 WP3 and ISO/IEC JT… [cited by applicant]
Lai et al., “Loop Filtering with Directional Features”, JCTVC-E288, Samsung Electronics Co., Ltd., Joint Collaborative Team on Video Coding (JCT-VC) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 5th Meeting: Geneva… [cited by applicant]
Li et al., “Complexity Reduction of an Adaptive Loop Filter Based on Local Homogeneity”, IEIE Transactions on Smart Processing and Computing, vol. 6, No. 2, Apr. 2017, pp. 93-101. [cited by applicant]
Lim, Sung-Chang et al., “Subsampled Sum-Modified-Laplacian for Adaptive Loop Filter in Versatile Video Coding”, 2020, 176330-176342. [cited by applicant]
McCann et al., “High Efficiency Video Coding (HEVC) Test Model 16 (HM 16) Improved Encoder Description”, JCTVC-S1002, Editors, Joint Collaborative Team on Video Coding (JCT-VC) of ITU-T SG16 WP3 and ISO/IEC JTC1/SC29/WG… [cited by applicant]
Miyazawa et al., “Complexity Reduction of In-Loop Filtering for Compressed Image Restoration in HEVC”, Picture Coding Symposium, May 7-9, 2012, pp. 413-416. [cited by applicant]
Miyazawa et al., “Reduction of the Number of Pixels Used in Adaptive Loop Filter”, JCTVC-G 446, Mitsubishi Electric Corporation, Joint Collaborative Team on Video Coding (JCT-VC) of ITU-T SG WP 3 and ISO/IEC JTC 1/SC 29… [cited by applicant]
Polin, Lai et al., “Computationally efficient adaptive loop filtering design with directional features for video coding”, 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Kyoto, Ja… [cited by applicant]
Vanam et al., “CE2: Subsampled Gradient Calculation for Highest Temporal Layer (Test 2.7.1)”, JVET-L0240, InterDigital Communications, Inc., Joint Video Experts (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, … [cited by applicant]
Xiu et al., “Description of SDR, HDR and 360° Video Coding Technology Proposal by InterDigital Communications and Dolby Laboratories”, JVET-J0015-V1, InterDigital Communications, Inc., Dolby Laboratories, Inc., Joint Vi… [cited by applicant]
Zheng et al., “Directional adaptive loop filter for video coding,” 18th IEEE International Conference on Image Processing, 2011, pp. 3501-3504. [cited by applicant]
Karczewicz et al., “Improvements on adaptive loop filter”, Document: JVET-B0060, Joint Video Exploration Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 2nd Meeting: San Diego, USA, Feb. 20-26, 2016, 5 pa… [cited by applicant]