IP Library › Granted Patent US 11,800,097
Granted Patent B2
US 11,800,097 · App. 17/517,878 · Granted Oct 24, 2023

Method for image processing and apparatus for implementing the same

Inventors: Anthony Nasrallah (Puteaux, FR); Thomas Guionnet (Rennes, FR); Mohsen Abdoli (Thorigne Fouillard, FR); Marco Cagnazzo (Gif sur Yvette, FR); Attilio Fiandrotti (Turin, IT)
Assignee: ATEME
H04N19/117G06N3/08G06T9/002H04N19/105H04N19/139H04N19/176
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,800,097
App. No.
17/517,878
Granted
Oct 24, 2023
Kind
B2
Abstract

A method of processing a first image in a first plurality of images, wherein the first image is divided into a plurality of pixel blocks, is proposed, which comprises, for a current block of the first image: selecting, in a set of a plurality of predefined interpolation filters, an interpolation filter based on a prediction of an interpolation filter determined by a supervised learning algorithm to which data related to the current block is input; and using the selected interpolation filter for calculating fractional pixel values in a second image of the plurality of images for a temporal prediction of pixels of the current block based on a reference block correlated to the current block in the second image, wherein the second image is distinct from the first image and was previously encoded according to an image encoding sequence for encoding the images of the plurality of images.

Claims (55)

1. A method of processing a first image in a first plurality of images, wherein the first image is divided into a plurality of pixel blocks, the method comprising, for a current block of the first image:

selecting, in a set of a plurality of predefined interpolation filters, a first interpolation filter based on a first prediction of an interpolation filter determined by a supervised learning algorithm to which data related to the current block is input; and

using the selected first interpolation filter for calculating fractional pixel values in a second image of the plurality of images for a temporal prediction of pixels of the current block based on a reference block correlated to the current block in the second image,

wherein the second image is distinct from the first image and was previously encoded according to an image encoding sequence for encoding the images of the plurality of images,

wherein the method further comprises a learning phase of a neural network performed on a second set of images,

wherein the learning phase comprises, for a current block of a current image of the second set of images:

selecting, in the set of the plurality of predefined interpolation filters, an interpolation filter based on a distortion criterion calculated for the current block; and

performing training of the neural network based on data related to the current block and the selected interpolation filter.

2. The method according to claim 1 , further comprising:

selecting, based on a first characteristic of the current block, a subset of interpolation filters in a set of predefined interpolation filters,

wherein the subset of interpolation filters is input to the supervised learning algorithm for determining the first prediction of the interpolation filter, and

wherein the first characteristic is obtainable based on the data related to the current block input to the supervised learning algorithm.

3. The method according to claim 1 , further comprising:

selecting, based on a second characteristic of the current block, the supervised learning algorithm.

4. The method according to claim 2 , wherein each of the first and second characteristics of the current block comprises one or more of a size of the current block, and a shape of the current block.

5. The method according to claim 1 , wherein the input data related to the current block comprise pixels of an application area comprising a set of at least one pixel in at least one pixel block of the first image,

wherein the at least one pixel block has already been processed according to a processing sequence defined for the first image.

6. The method according to claim 1 , wherein the input data related to the current block comprise pixels from a motion compensated block in the second image.

7. The method according to claim 1 , wherein the determining of the first prediction of interpolation filter is based on an identifier in the set of predefined interpolation filters output by the supervised learning algorithm.

8. The method according to claim 1 , wherein the supervised learning algorithm is a gradient-based learning algorithm.

9. The method according to claim 1 , wherein the set of predefined interpolation filters comprises a plurality of low-pass filters with respective cut-off frequencies.

10. The method according to claim 1 , further comprising:

selecting, in the set of the plurality of predefined interpolation filters, a second interpolation filter based on a second prediction of an interpolation filter determined by the supervised learning algorithm to which data related to the current block is input;

using the selected second interpolation filter for calculating fractional pixel values in a third image of the plurality of images in a second direction for a temporal prediction of pixels of the current block in the second direction based on a reference block correlated to the current block in the third image,

wherein the third image is distinct from the first image and was previously encoded according to the image encoding sequence,

wherein the selected first interpolation filter is used for calculating the fractional pixel values in the second image in a first direction for a temporal prediction of pixels of the current block in the first direction, and wherein the first direction is distinct from the second direction.

11. The method according to claim 1 , wherein the supervised learning algorithm is a convolutional neural network learning algorithm.

12. An apparatus comprising a processor and a memory operatively coupled to the processor, wherein the apparatus is configured to perform a method of processing a first image in a first plurality of images, wherein the first image is divided into a plurality of pixel blocks, and the method comprises, for a current block of the first image:

selecting, in a set of a plurality of predefined interpolation filters, a first interpolation filter based on a first prediction of an interpolation filter determined by a supervised learning algorithm to which data related to the current block is input; and

using the selected first interpolation filter for calculating fractional pixel values in a second image of the plurality of images for a temporal prediction of pixels of the current block based on a reference block correlated to the current block in the second image,

wherein the second image is distinct from the first image and was previously encoded according to an image encoding sequence for encoding the images of the plurality of images,

wherein the method further comprises a learning phase of a neural network performed on a second set of images,

wherein the learning phase comprises, for a current block of a current image of the second set of images:

selecting, in the set of the plurality of predefined interpolation filters, an interpolation filter based on a distortion criterion calculated for the current block; and

performing training of the neural network based on data related to the current block and the selected interpolation filter.

13. The apparatus according to claim 12 , wherein the apparatus is further configured to perform the steps:

selecting, based on a first characteristic of the current block, a subset of interpolation filters in a set of predefined interpolation filters,

wherein the subset of interpolation filters is input to the supervised learning algorithm for determining the first prediction of the interpolation filter, and

wherein the first characteristic is obtainable based on the data related to the current block input to the supervised learning algorithm.

14. The apparatus according to claim 12 , wherein the apparatus is further configured to perform the step of: selecting, based on a second characteristic of the current block, the supervised learning algorithm.

15. The apparatus according to claim 13 , wherein each of the first and second characteristics of the current block comprises one or more of a size of the current block, and a shape of the current block.

16. A non-transitory computer-readable medium encoded with executable instructions which, when executed, causes an apparatus comprising a processor operatively coupled with a memory, to perform a method of processing a first image in a first plurality of images, wherein the first image is divided into a plurality of pixel blocks, and the method comprises, for a current block of the first image:

selecting, in a set of a plurality of predefined interpolation filters, a first interpolation filter based on a first prediction of an interpolation filter determined by a supervised learning algorithm to which data related to the current block is input; and

using the selected first interpolation filter for calculating fractional pixel values in a second image of the plurality of images for a temporal prediction of pixels of the current block based on a reference block correlated to the current block in the second image,

wherein the second image is distinct from the first image and was previously encoded according to an image encoding sequence for encoding the images of the plurality of images,

wherein the method further comprises a learning phase of a neural network performed on a second set of images,

wherein the learning phase comprises, for a current block of a current image of the second set of images:

selecting, in the set of the plurality of predefined interpolation filters, an interpolation filter based on a distortion criterion calculated for the current block; and

performing training of the neural network based on data related to the current block and the selected interpolation filter.

17. The non-transitory computer-readable medium according to claim 16 , wherein the executable instructions further cause the apparatus to perform the steps:

selecting, based on a first characteristic of the current block, a subset of interpolation filters in a set of predefined interpolation filters,

wherein the subset of interpolation filters is input to the supervised learning algorithm for determining the first prediction of the interpolation filter, and

wherein the first characteristic is obtainable based on the data related to the current block input to the supervised learning algorithm.

18. The non-transitory computer-readable medium according to claim 16 , wherein the executable instructions further cause the apparatus to perform the step: selecting, based on a second characteristic of the current block, the supervised learning algorithm.

19. The non-transitory computer-readable medium according to claim 17 , wherein each of the first and second characteristics of the current block comprises one or more of a size of the current block, and a shape of the current block.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 13, 2022
From: NASRALLAH, ANTHONY; GUIONNET, THOMAS; ABDOLI, MOHSEN; CAGNAZZO, MARCO; FIANDROTTI, ATTILIO
To: ATEME
Reel/Frame 059898/0488 →
Priority Claims (1)
EP 20306318 · Nov 3, 2020 · regional
Continuity (1)
Related Publication 20220141460A1 · May 5, 2022