Method, apparatus and computer program product for end-to-end learned predictive coding of media frames

1 . An apparatus comprising:

at least one processor; and

at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus at least to perform:

encode or decode one or more media frames based on at least one neural network;

wherein an inter-frame codec is applied to at least one media frame of the one or more media frames;

wherein the at least one media frame is referred to as a target media frame;

wherein a first decoded reference frame and a second decoded reference frame refers to reference frames for the target media frame; and

wherein the inter-frame codec comprises:

an inter-frame encoder block or circuit, wherein the inter-frame encoder block or circuit is caused to encode the target media frame and obtain an encoded target frame; and

an inter-frame decoder block or circuit to generate a decoded target frame based on the encoded target frame;

generate a first set of multi-scale features based on the first decoded reference frame, the second decoded reference frame, and the target media frame;

perform at least one of aggregating or converting multi-scale features of the first set of multi-scale features to a latent representation, wherein the latent representation comprises information about motion between the target media frame and one or more reference frames;

determine a first temporal level for the first decoded reference frame;

wherein the first temporal level determined for the first decoded reference frame is determined based on the first decoded reference frame being temporally before the target media frame;

select a first index for the first decoded reference frame based on the first temporal level determined for the first decoded reference frame;

determine a second temporal level for the second decoded reference frame;

wherein the second temporal level determined for the second decoded reference frame is determined based on the second decoded reference frame being temporally after the target media frame;

select a second index for the second decoded reference frame based on the second temporal level determined for the second decoded reference frame; and

use the first index selected for the first decoded reference frame and the second index selected for the second decoded reference frame to encode or decode the target media frame;

wherein the target media frame comprises a bidirectionally predicted media frame.

2 . The apparatus claim 1 , wherein the apparatus further comprises one or more feature pyramid net blocks or circuits, and wherein the one or more feature pyramid net blocks or circuits are caused to generate the first set of multi-scale features based on the first decoded reference frame, the second decoded reference frame, and the target media frame.

3 . The apparatus of the claim 2 , wherein the apparatus further comprises a bridge net block or circuit, and wherein the bridge net block or circuit is caused to aggregate and/or convert the first set of multi-scale features to the latent representation, wherein the latent representation comprises information about motion-prediction residual.

4 . The apparatus of claim 3 , wherein the apparatus further comprises an entropy codec block or circuit, and wherein the entropy codec block or circuit comprises an entropy encoder block or circuit that is caused to quantize, compress and convert the latent representation to a compressed bitstream and wherein the entropy codec block or circuit further comprises an entropy decoder block or circuit that is caused to decompress and dequantize the compressed bitstream to generate a reconstructed latent representation.

5 . The apparatus of claim 4 , wherein the apparatus further comprises at least one entropy feature pyramid net block or circuit, and wherein the at least one entropy feature pyramid net block or circuit is caused to generate one or more features from the first decoded reference frame and the second decoded reference frame, wherein the one or more features are used to enhance a context, wherein the context is used by a probability model to obtain an estimated probability distribution of the latent representation, and wherein the estimated probability distribution is used by the entropy codec block or circuit.

6 . The apparatus of claim 4 , wherein the apparatus further comprises a feature pyramid decoder, and wherein the feature pyramid decoder is caused to generate a second set of multi-scale features and a residual output based on the reconstructed latent representation.

7 . The apparatus of claim 6 , wherein the apparatus further comprises a first motion estimation block or circuit and a second motion estimation block or circuit, wherein the first motion estimation block or circuit is caused to estimate a first motion based at least on the second set of multi-scale features or on features extracted from the first decoded reference frame to obtain a first estimated motion, and wherein the second motion estimation block or circuit is caused to estimate a second motion based at least on the second set of multi-scale features or on features extracted from a second decoded reference frame to obtain a second estimated motion.

8 . The apparatus of claim 7 , wherein the apparatus further comprises one or more frame prediction blocks or circuits, and wherein the one or more frame prediction blocks or circuits are caused to use the first estimated motion and the second estimated motion to warp a first reference frame to generate a first prediction of the target media frame and a second reference frame to generate a second prediction of the target media frame.

9 . The apparatus of claim 8 , wherein the apparatus further comprises a combiner block or circuit, and wherein the combiner block or circuit is caused to generate a final prediction of the target media frame based at least on one or more of the following: the first prediction of the target media frame, the second prediction of the target media frame, one or more decoded intra frames, a latent representation of one or more intra frames, or the residual output.

10 . The apparatus claim 1 , wherein a distance between the target media frame and one of the first decoded reference frame or the second decoded reference frame, or information derived from the distance between the target media frame and one of the first decoded reference frame or the second decoded reference frame, is input to one or more neural networks of the inter-frame encoder block or circuit and/or the inter-frame decoder block or circuit.

11 . A method comprising:

encoding or decoding one or more media frames based on at least one neural network;

wherein an inter-frame codec is applied to at least one media frame of the one or more media frames;

wherein the at least one media frame is referred to as a target media frame;

wherein a first decoded reference frame and a second decoded reference frame refers to reference frames for the target media frame; and

wherein the inter-frame codec comprises:

an inter-frame encoder block or circuit, wherein the inter-frame encoder block or circuit is caused to encode the target media frame and obtain an encoded target frame; and

an inter-frame decoder block or circuit to generate a decoded target frame based on the encoded target frame;

generating a first set of multi-scale features based on the first decoded reference frame, the second decoded reference frame, and the target media frame;

performing at least one of aggregating or converting multi-scale features of the first set of multi-scale features to a latent representation, wherein the latent representation comprises information about motion between the target media frame and one or more reference frames;

determining a first temporal level for the first decoded reference frame;

wherein the first temporal level determined for the first decoded reference frame is determined based on the first decoded reference frame being temporally before the target media frame;

selecting a first index for the first decoded reference frame based on the first temporal level determined for the first decoded reference frame;

determining a second temporal level for the second decoded reference frame;

wherein the second temporal level determined for the second decoded reference frame is determined based on the second decoded reference frame being temporally after the target media frame;

selecting a second index for the second decoded reference frame based on the second temporal level determined for the second decoded reference frame; and

using the first index selected for the first decoded reference frame and the second index selected for the second decoded reference frame to encode or decode the target media frame;

wherein the target media frame comprises a bidirectionally predicted media frame.

12 . The method of claim 11 , wherein the latent representation comprises information about motion-prediction residual.

13 . The method of claim 12 further comprising:

quantizing, compressing and converting the latent representation to a compressed bitstream; and

decompressing and dequantizing the compressed bitstream to generate a reconstructed latent representation.

14 . The method of claim 13 further comprising generating one or more features from the first decoded reference frame and the second decoded reference frame, wherein the one or more features are used to enhance a context, and wherein the context is used by a probability model to obtain an estimated probability distribution of the latent representation, and wherein the estimated probability distribution is used by an entropy codec block or circuit.

15 . The method of claim 13 further comprising generating a second set of multi-scale features and a residual output based on the reconstructed latent representation.

16 . The method of claim 15 further comprising:

estimating a first motion based at least on the second set of multi-scale features or on features extracted from the first decoded reference frame to obtain a first estimated motion; and

estimating a second motion based at least on the second set of multi-scale features or on features extracted from a second decoded reference frame to obtain a second estimated motion.

17 . The method of claim 16 further comprising:

using the first estimated motion and the second estimated motion to warp a first reference frame to generate a first prediction of the target media frame and a second reference frame to generate a second prediction of the target media frame; and

generating a final prediction of the target media frame based at least on one or more of the following: the first prediction of the target media frame, the second prediction of the target media frame, one or more decoded intra frames, a latent representation of one or more intra frames, or the residual output.

18 . The method claim 11 , wherein a distance between the target media frame and one of the first decoded reference frame or the second decoded reference frame, or information derived from the distance between the target media frame and one of the first decoded reference frame or the second decoded reference frame, is input to one or more neural networks of the inter-frame encoder block or circuit and/or the inter-frame decoder block or circuit.