IP Library › Granted Patent US 12,627,836
Granted Patent B2
US 12,627,836 · App. 17/767,989 · Granted May 12, 2026

Deep intra predictor generating side information

Inventors: Thierry Dumas (Rennes, FR); Franck Galpin (Thorigne-Fouillard, FR); Jean Begaint (Menlo Park, CA); Fabien Racape (San Francisco, CA)
Assignee: InterDigital Madison Patent Holdings, SAS
H04N19/593H04N19/11H04N19/172H04N19/176H04N19/186H04N19/42H04N19/70
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,627,836
App. No.
17/767,989
Granted
May 12, 2026
Kind
B2
Abstract

At least a method and an apparatus are presented for efficiently encoding or decoding video. For example, an intra prediction of an image block and side information are determined using at least one neural network from a context comprising pixels surrounding the image block. The side information allows a decoder to determine the intra prediction and is signaled for the decoding.

Claims (54)

1 . A method comprising:

decoding from a bitstream, for a luma block being decoded in a picture of a video, a side information for a neural network intra prediction, the side information and the neural network intra prediction having been generated at encoding by applying a neural network to the luma block and to a context surrounding the luma block, the side information comprising neural network feature values encoded into the bitstream;

determining, for the luma block being decoded, a luma component of the neural network intra prediction by applying a neural network to the side information and to the context surrounding the luma block, wherein the side information guides the neural network intra prediction at the decoding; and

decoding the luma block using the determined neural network intra prediction.

2 . The method of claim 1 , further comprising:

decoding, for a chroma block being decoded in the picture of the video, a side information for the neural network intra prediction, the side information and the neural network intra prediction having been generated at encoding by applying the neural network to a context surrounding the chroma block and the context surrounding the luma block; and

determining, for the chroma block, a chroma component of the neural network intra prediction by applying a neural network to the context surrounding the chroma block, the context surrounding the luma block, and the side information.

3 . The method of claim 1 , further comprising:

decoding, for a chroma block being decoded in the picture of the video, a side information for the neural network intra prediction, the side information and the neural network intra prediction having been generated at encoding by applying the neural network to a context surrounding a luma block that is collocated with the chroma block; and

determining, for the chroma block, a chroma component of the neural network intra prediction by applying a neural network to the context surrounding the luma block that is collocated with the chroma block and the side information.

4 . The method of claim 1 further comprising:

decoding a syntax element indicating that a neural network-based intra prediction is used for intra prediction of the luma block.

5 . An apparatus comprising one or more processors, wherein the one or more processors are configured to:

decode from a bitstream, for a luma block being decoded in a picture of a video, a side information for a neural network intra prediction, the side information and the neural network intra prediction having been generated at encoding by applying a neural network to the luma block and to a context surrounding the luma block, the side information comprising neural network feature values encoded into the bitstream;

determine, for the luma block being decoded, a luma component of the neural network intra prediction by applying a neural network to the side information and to the context surrounding the luma block, wherein the side information guides the neural network intra prediction at the decoding; and

decode the luma block using the determined neural network intra prediction.

6 . The apparatus of claim 5 , wherein the one or more processors are configured to:

decode, for a chroma block being decoded in the picture of the video, a side information for the neural network intra prediction, the side information and the neural network intra prediction having been generated at encoding by applying the neural network to a context surrounding the chroma block and a context surrounding a luma block that is collocated with the chroma block; and

determine, for the chroma block, a chroma component of the neural network intra prediction by applying a neural network to the context surrounding the chroma block, the context surrounding the luma block, and the side information.

7 . The apparatus of claim 5 , wherein the one or more processors are configured to:

decode, for a chroma block being decoded in the picture of the video, a side information for the neural network intra prediction, the side information and the neural network intra prediction having been generated at encoding by applying a neural network to a context surrounding a luma block that is collocated with the chroma block;

determine, for the chroma block, a chroma component of the neural network intra prediction and side information by applying a neural network to the a context surrounding the luma block and the side information.

8 . The apparatus of claim 5 , wherein the one or more processors are configured to:

decode a syntax element indicating that a neural network-based intra prediction is used for intra prediction of the luma block.

9 . A method for video encoding, comprising:

determining, from a luma block being encoded in a picture of a video, by applying a neural network to the luma block and to a context surrounding the luma block, a luma component of a neural network intra prediction and a side information to guide neural network intra prediction at decoding, the side information comprising neural network feature values;

encoding the luma block based on the neural network intra prediction; and

encoding the side information into a bitstream.

10 . The method of claim 9 , further comprising:

determining, for a chroma block collocated with the luma block being encoded in the picture of the video, a chroma component of the neural network intra prediction and side information by applying the neural network to the chroma block and to the context surrounding the luma block; and

encoding the chroma block based on the determined neural network intra prediction.

11 . The method of claim 9 , further comprising:

determining, for a chroma block being encoded in the picture of the video, a chroma component of the neural network intra prediction and side information by applying the neural network to a context surrounding the chroma block and the context surrounding the luma block; and

encoding the chroma block based on the determined neural network intra prediction.

12 . The method of claim 9 , further comprising:

determining, for a chroma block being encoded in the picture of the video, a chroma component of the neural network intra prediction and side information by applying the neural network to a context surrounding a luma block that is collocated with the chroma block; and

encoding the chroma block based on the determined neural network intra prediction.

13 . The method of claim 9 , further comprising:

encoding a syntax element indicating that a neural network-based intra prediction is used for intra prediction of the luma block.

14 . An apparatus for video encoding, comprising one or more processors, wherein the one or more processors are configured to:

determine, from a luma block being encoded in a picture of a video, by applying a neural network to the luma block and to a context surrounding the luma block, a luma component of a neural network intra prediction and a side information to guide neural network intra prediction at decoding, the side information comprising neural network feature values;

encode the luma block based on the neural network intra prediction; and

encode the side information into a bitstream.

15 . The apparatus of claim 14 , wherein the one or more processors are configured to:

determine, for a chroma block collocated with the luma block being encoded in the picture of the video, a chroma component of the neural network intra prediction and side information by applying the neural network to the chroma block and to the context surrounding the luma block; and

encode the chroma block based on the determined neural network intra prediction.

16 . The apparatus of claim 14 , wherein the one or more processors are configured to:

determine, for a chroma block being encoded in the picture of the video, a chroma component of the neural network intra prediction and side information by applying the neural network to a context surrounding the chroma block and the context surrounding the luma block; and

encode the chroma block based on the determined neural network intra prediction.

17 . The apparatus of claim 14 , wherein the one or more processors are configured to:

determine, for a chroma block being encoded in the picture of the video, a chroma component of the neural network intra prediction and side information by applying the neural network to a context surrounding a luma block that is collocated with the chroma block; and

encode the chroma block based on the determined neural network intra prediction.

18 . The apparatus of claim 14 , wherein the one or more processors are configured to:

encode a syntax element indicating that a neural network-based intra prediction is used for intra prediction of the luma block.

Assignments (3)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Nov 6, 2023
From: INTERDIGITAL CE PATENT HOLDINGS, SAS
To: INTERDIGITAL MADISON PATENT HOLDINGS, SAS
Reel/Frame 065465/0293 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 1, 2023
From: INTERDIGITAL VC HOLDINGS FRANCE, SAS
To: INTERDIGITAL CE PATENT HOLDINGS, SAS
Reel/Frame 064460/0921 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 10, 2022
From: DUMAS, THIERRY; GALPIN, FRANCK; BEGAINT, JEAN; RACAPE, FABIEN
To: INTERDIGITAL VC HOLDINGS FRANCE, SAS
Reel/Frame 059880/0531 →
Priority Claims (1)
EP 19306330 · Oct 11, 2019 · regional
Continuity (1)
Related Publication 20230254507A1 · Aug 10, 2023
References Cited (31)
US 20130064292A1 · Song et al. · 2013 [cited by applicant]
US 20190246102A1 · Cho et al. · 2019 [cited by applicant]
US 20200137384A1 · Kwong · 2020 [cited by examiner]
US 20200186809A1 · Mukherjee · 2020 [cited by examiner]
US 20200244955A1 · Pfaff et al. · 2020 [cited by applicant]
US 20200280717A1 · Li · 2020 [cited by examiner]
US 20210014531A1 · Pfaff · 2021 [cited by examiner]
US 20210021823A1 · Na · 2021 [cited by examiner]
US 20210211709A1 · Zhang · 2021 [cited by examiner]
US 20210235104A1 · Huo · 2021 [cited by examiner]
US 20210314562A1 · Kang · 2021 [cited by examiner]
US 20220191548A1 · Yu · 2022 [cited by examiner]
CN 102907100A · 2013 [cited by applicant]
GB 2575628A · 2020 [cited by applicant]
TW 201924331A · 2019 [cited by applicant]
WO 2019011968A1 · 2019 [cited by applicant]
WO 2019029560A1 · 2019 [cited by applicant]
WO 2019194425A1 · 2019 [cited by applicant]
Anonymous, “Reference software for ITU-T H.265 high efficiency video coding”, International Telecommunication Union, ITU-T Telecommunication Standardization Sector of ITU, Series H: Audiovisual and Multimedia Systems, I… [cited by applicant]
Hu et al., “Progressive Spatial Recurrent Neural Network for Intra Prediction”, Institute of Electrical and Electronics Engineers (IEEE), IEEE Transactions on MultiMedia, vol. 21, Issue: 12, Dec. 2019, 14 pages. [cited by applicant]
Anonymous, “Versatile Video Coding”, International Telecommunication Union, ITU-T Telecommunication Standardization Sector of ITU, Series H: Audiovisual and Multimedia Systems, Infrastructure of audiovisual services—Cod… [cited by applicant]
Anonymous, “Series H: Audiovisual and Multimedia Systems, Infrastructure of audiovisual services—Coding of moving video—Information Technology—Generic coding of moving pictures and associated audio information: Video”, … [cited by applicant]
Minnen et al., “Spatially Adaptive Image Compression using a Tiled Deep Network”, Institute of Electrical and Electronics Engineers (IEEE), 2017 IEEE International Conference on Image Processing (ICIP), Beijing, China, … [cited by applicant]
Brand et al., “Intra Frame Prediction for Video Coding Using a Conditional Autoencoder Approach”, Institute of Electrical and Electronics Engineers (IEEE), 2019 Picture Coding Symposium (PCS), Ningbo, China, Nov. 15, 20… [cited by applicant]
Anonymous, “Transmission of Non-Telephone Signals: Information Technology—Generic Coding of Moving Pictures and Associated Audio Information: Systems”, International Telecommunication Union, ITU-T Telecommunication Stan… [cited by applicant]
Pfaff et al., Intra Prediction Modes based on Neural Networks, Joint Video Exploration Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, Document: JVET-J0037-v1, 10th Meeting, San Diego, California, USA, Ap… [cited by applicant]
Pathak et al., “Context Encoders: Feature Learning by Inpainting”, Institute of Electrical and Electronics Engineers (IEEE), 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, Nevada, USA… [cited by applicant]
Pfaff et al., CE3: Affine linear weighted intra prediction (CE3-4.1, CE3-4.2), Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, Document: JVET-N0217, 14th Meeting, Geneva, Switzerland, … [cited by applicant]
Wang et al., “Enhancing HEVC Spatial Prediction by Context-Based Learning”, Institute of Electrical and Electronics Engineers (IEEE), ICASSP 2019—2019 IEEE International Conference on Acoustics, Speech and Signal Proces… [cited by applicant]
Li et al., “Fully-Connected Network-Based Intra Prediction for Image Coding”, Institute of Electrical and Electronics Engineers (IEEE), IEEE Transactions on Image Processing, vol. 27, No. 7, Jul. 2018, 12 pages. [cited by applicant]
Dumas et al., “Context-Adaptive Neural Network-Based Prediction for Image Compression”, Institute of Electrical and Electronics Engineers (IEEE), IEEE Transactions on Image Processing, vol. 29, Aug. 16, 2019, 16 pages. [cited by applicant]