Granted Patent B2

US 12,321,870 · App. 17/457,136 · Granted Jun 3, 2025

Inventors: Nannan Zou (Tampere, FI); Francesco Cricrì (Tampere, FI); Honglei Zhang (Tampere, FI); Hamed Rezazadegan Tavakoli (Espoo, FI); Jani Lainema (Tampere, FI); Miska Matias Hannuksela (Tampere, FI)

Assignee: Nokia Technologies Oy

G06N7/01G06V10/44G06V10/467G06V10/84H04N19/176H04N19/61

Abstract

Claims (127)

1. An apparatus comprising

at least one processor; and

at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus at least to:

performing, at an encoder, first overfitting of a probability model by performing at least one or more training operations applied to the probability model to obtain an overfitted probability model, wherein one or more parameters of the probability model are trained via the first overfitting;

using, at the encoder, the overfitted probability model to provide probability estimates to a lossless codec or a substantially lossless codec for encoding data;

outputting the encoded data to become part of a bitstream; and

signaling, by the encoder in the bitstream, information to a decoder to perform a second overfitting of a probability model used at the decoder.

2. The apparatus of claim 1 , wherein an input data is used to overfit the probability model at the encoder, and wherein the input data is sampled from or comprised in data that was previously encoded by the lossless codec or the substantially lossless codec.

3. The apparatus of claim 2 , wherein the input data comprises one or more of the following:

one or more previously encoded frames;

one or more pixels of a previously encoded frame;

one or more pixels of a currently encoded frame;

one or more pixels that have been encoded in past predetermined time, wherein the one or more pixels are part of a frame;

one or more previously encoded latent tensors or feature tensors;

one or more elements of a previously encoded latent tensor or feature tensor;

one or more elements of a currently encoded latent tensor or feature tensor; or

one or more elements that have been encoded in past predetermined time, wherein the one or more elements are part of a latent tensor or a feature tensor.

4. The apparatus of claim 2 , wherein-the-data, from which the input data is sampled or comprised in, comprises an overfitting media item, and wherein the overfitting media item comprises previously encoded data, and wherein the previously encoded data comprises one or more of following:

a portion of a frame of video, or features extracted from the frame;

a downsampled version of the frame, or features extracted from the downsampled version of the frame;

a set of portions of the frame, or features extracted from the set of the portions of the frame;

a set of portions of a set of frames, or features extracted from the set of the portions of the set of the frames;

the frame of the video, or features extracted from the frame of the video;

a set of frames of the video or features extracted from the set of the frames of the video; or

a set of videos, or features extracted from the set of the videos.

5. The apparatus of claim 1 , wherein the apparatus is further caused to use the overfitted probability model at the encoder or a previously available probability model at the encoder to encode data that follow an overfitted media item or a portion of the data that follows the overfitted media item in decoding order, wherein data that follow the overfitted media item or a portion of the data that follows the overfitted media item comprises an inference media item.

6. The apparatus of claim 1 , wherein the information signaled to the decoder further comprises details about one or more of the following:

a version of the probability model to be used when the second overfitting is not to be performed;

a version of the probability model to be used as a base probability model for performing the second overfitting;

an overfitting media item to be used for performing the second overfitting, wherein the information further comprises a type of the overfitting media item and coordinates to uniquely identify a specific overfitting media item as the overfitting media item;

data to be decoded by using the probability model referred in the signal; or

training, wherein the training is part of the second overfitting, and wherein details about the training comprise a number of epochs or iterations, a learning rate, an optimizer routine, a loss function, a loss function schedule, a data preprocessing information, any seeds for pseudo-random processes, or any other information which is needed by the decoder to perform the second overfitting in same or substantially same way as performed by the encoder.

7. An apparatus comprising

at least one processor; and

at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus at least to:

receiving, from a bitstream at a decoder from an encoder, encoded data;

receiving, in the bitstream, a signal comprising information for indicating to perform an overfitting of a probability model at the decoder;

performing, based on the signal received, the overfitting of the probability model at the decoder by performing one or more training operations applied to the probability model to obtain an overfitted probability model, wherein one or more parameters of the probability model are trained via the overfitting;

using, at the decoder, the overfitted probability model to provide probability estimates to a lossless codec or a substantially lossless codec for decoding the encoded data to form decoded data; and

outputting the decoded data.

8. The apparatus of claim 7 , wherein the information further comprises details about one or more of the following:

a version of the probability model to be used when the overfitting is not be performed;

a version of the probability model to be used as a base probability model for performing the overfitting;

an overfitting media item to be used for performing the overfitting, wherein the information further comprises a type of overfitting media item and coordinates to uniquely identify a specific overfitting media item;

data to be decoded by using the probability model referred in the signal; or

training, wherein the training is part of the overfitting, and wherein details about the training comprise a number of epochs or iterations, a learning rate, an optimizer routine, a loss function, a loss function schedule, a data preprocessing information, any seeds for pseudo-random processes, or any other information which is needed by the decoder to perform the overfitting in a same or substantially a same way as performed by the encoder.

9. The apparatus of claim 7 , wherein an input data is used to overfit the probability model, and wherein the input data is sampled from or comprised in data that was previously decoded by the lossless codec or the substantially lossless codec.

10. The apparatus of claim 9 , wherein the input data comprises one or more of the following:

one or more previously decoded frames;

one or more pixels of a previously decoded frame;

one or more pixels of a currently decoded frame;

one or more pixels that have been decoded in past predetermined time, wherein the one or more pixels are part of a frame;

one or more previously decoded latent tensors or feature tensors;

one or more elements of a previously decoded latent tensor or feature tensor;

one or more elements of a currently decoded latent tensor or feature tensor; or

one or more elements that have been decoded in past predetermined time, wherein the one or more elements are part of a latent tensor or a feature tensor.

11. The apparatus of claim 9 , wherein data, from which the input data is sampled or comprised in, comprises an overfitting media item, and wherein the overfitting media item comprises previously decoded data, and wherein the previously decoded data comprises one or more of following:

a portion of a frame of video, or features extracted from the frame;

a downsampled version of the frame, or features extracted from the downsampled version of the frame;

a set of portions of the frame, or features extracted from the set of the portions of the frame;

a set of portions of a set of frames, or features extracted from the set of portions of the set of the frames;

the frame of the video, or features extracted from the frame of the video;

a set of frames of the video or features extracted from the set of the frames of the video; or

a set of videos, or features extracted from the set of the videos.

12. The apparatus of claim 7 , wherein the apparatus is further caused to use the overfitted probability model or previously available probability model to decode data that follow an overfitted media item or a portion of the data that follows the overfitted media item in decoding order, wherein data that follow the overfitted media item or a portion of the data that follows the overfitted media item comprises an inference media item.

13. A method comprising:

performing, at an encoder, first overfitting of a probability model by performing at least one or more training operations applied to the probability model, wherein one or more parameters of the probability model are trained via the first overfitting;

using, at the encoder, the probability model overfit at the encoder to provide probability estimates to a lossless codec or a substantially lossless codec for encoding data;

outputting the encoded data to become part of a bitstream; and

signaling, by the encoder in the bitstream, information to a decoder to perform a second overfitting of a probability model used at the decoder.

14. The method of claim 13 , wherein an input data is used to overfit the probability model at the encoder, and wherein the input data is sampled from or comprised in data that was previously encoded by the lossless codec or the substantially lossless codec.

15. The method of claim 14 , wherein the input data comprises one or more of the following:

one or more previously encoded frames;

one or more pixels of a previously encoded frame;

one or more pixels of a currently encoded frame;

one or more pixels that have been encoded in past predetermined time, wherein the one or more pixels are part of a frame;

one or more previously encoded latent tensors or feature tensors;

one or more elements of a previously encoded latent tensor or feature tensor;

one or more elements of a currently encoded latent tensor or feature tensor; or

one or more elements that have been encoded in past predetermined time, wherein the one or more elements are part of a latent tensor or a feature tensor.

16. The method of claim 14 , wherein data, from which the input data is sampled or comprised in, comprises an overfitting media item, and wherein the overfitting media item comprises previously encoded data, and wherein the previously encoded data comprises one or more of following:

a portion of a frame of video, or features extracted from the frame;

a downsampled version of the frame, or features extracted from the downsampled version of the frame;

a set of portions of the frame, or features extracted from the set of the portions of the frame;

a set of portions of a set of frames, or features extracted from the set of the portions of the set of the frames;

the frame of the video, or features extracted from the frame of the video;

a set of frames of the video or features extracted from the set of the frames of the video; or

a set of videos, or features extracted from the set of the videos.

17. The method of claim 13 , further comprising using the probability model overfit at the encoder or previously available probability model at the encoder to encode data that follow an overfitted media item or a portion of the data that follows the overfitted media item in decoding order, wherein data that follow the overfitted media item or a portion of the data that follows the overfitted media item comprises an inference media item.

18. The method of claim 13 , wherein the information signaled to the decoder further comprises details about one or more of the following:

a version of the probability model to be used when the second overfitting is not to be performed;

a version of the probability model to be used as a base probability model for performing the second overfitting;

an overfitting media item to be used for performing the second overfitting, wherein the information further comprises a type of the overfitting media item and coordinates to uniquely identify a specific overfitting media item as the overfitting media item;

data to be decoded by using the probability model referred in the information signaled to the decoder; or

training, wherein the training is part of the second overfitting, and wherein details about the training comprise a number of epochs or iterations, a learning rate, an optimizer routine, a loss function, a loss function schedule, a data preprocessing information, any seeds for pseudo-random processes, or any other information which is needed by the decoder to perform the second overfitting in same or substantially same way as performed by the encoder.

19. A method comprising:

receiving, from a bitstream at a decoder from an encoder, encoded data;

receiving, in the bitstream, a signal comprising information for indicating to perform overfitting of a probability model at the decoder;

performing, based on the signal received, the overfitting of the probability model at the decoder to obtain an overfitted probability model, wherein one or more parameters of the probability model are trained via the overfitting; and

using, at the decoder, the overfitted probability model to provide probability estimates to a lossless codec or a substantially lossless codec for decoding the encoded data or to form decoded data and

outputting the decoded data.

20. The method of claim 19 , wherein the information further comprises details about one or more of the following:

a version of the probability model to be used when the overfitting is not to be performed;

a version of the probability model to be used as a base probability model for performing the overfitting;

an overfitting media item to be used for performing the overfitting, wherein the information further comprises a type of overfitting media item and coordinates to uniquely identify a specific overfitting media item;

data to be decoded by using the probability model referred in the signal; or

training, wherein the training is part of the overfitting, and wherein details about the training comprise a number of epochs or iterations, a learning rate, an optimizer routine, a loss function, a loss function schedule, a data preprocessing information, any seeds for pseudo-random processes, or any other information which is needed by the decoder to perform the overfitting in same or substantially same way as performed by the encoder.

21. The method of claim 19 , wherein an input data is used to overfit the probability model, and wherein the input data is sampled from or comprised in data that was previously decoded by the lossless codec or the substantially lossless codec.

22. The method of claim 21 , wherein the input data comprises one or more of the following:

one or more previously decoded frames;

one or more pixels of a previously decoded frame;

one or more pixels of a currently decoded frame;

one or more pixels that have been decoded in past predetermined time, wherein the one or more pixels are part of a frame;

one or more previously decoded latent tensors or feature tensors;

one or more elements of a previously decoded latent tensor or feature tensor;

one or more elements of a currently decoded latent tensor or feature tensor; or

one or more elements that have been decoded in past predetermined time, wherein the one or more elements are part of a latent tensor or a feature tensor.

23. The method of claim 21 , wherein data, from which the input data is sampled or comprised in, comprises an overfitting media item, and wherein the overfitting media item comprises previously decoded data, and wherein the previously decoded data comprises one or more of following:

a portion of a frame of video, or features extracted from the frame;

a downsampled version of the frame, or features extracted from the downsampled version of the frame;

a set of portions of the frame, or features extracted from the set of the portions of the frame;

a set of portions of a set of frames, or features extracted from the set of the portions of the set of the frames;

the frame of the video, or features extracted from the frame of the video;

a set of frames of the video or features extracted from the set of the frames of the video;

or

a set of videos, or features extracted from the set of the videos.

24. The method of claim 19 further comprising using the overfitted probability model or previously available probability model to decode data that follow an overfitted media item or a portion of the data that follows the overfitted media item in decoding order, wherein data that follow the overfitted media item or a portion of the data that follows the overfitted media item comprises an inference media item.

Assignments (1)

Continuity (1)

References Cited (22)

Cited By (1)

US 12,744,915