Granted Patent B2

US 12,536,711 · App. 18/053,253 · Granted Jan 27, 2026

Inventors: Francesco Cricrì (Tampere, FI); Honglei Zhang (Tampere, FI); Miska Matias Hannuksela (Tampere, FI); Hamed Rezazadegan Tavakoli (Espoo, FI); Nam Hai Le (Tampere, FI); Ramin Ghaznavi Youvalari (Tampere, FI); Jukka Ahonen (Tampere, FI); Emre Baris Aksu (Tampere, FI)

Assignee: Nokia Technologies Oy

G06T9/002G06T3/4046

Abstract

Claims (64)

1 . An apparatus comprising:

at least one processor; and

at least one memory comprising computer program code; wherein the at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus at least to iteratively perform following until a stopping criterion is met:

provide a finetuning driving content (FDC) or a content derived from FDC to a decoder side neural network (DSNN);

compute an output of the DSNN as a processed FDC;

compute a loss based on the processed FDC and an approximated ground truth data (AGT) associated with the FDC;

compute an update to the DSNN;

apply the computed update to the DSNN;

run a loss proxy neural network for one or more input frames; and

encode an output of the loss proxy neural network, representing the AGT, as metadata.

2 . The apparatus of claim 1 , wherein to compute the update to the DSNN, the apparatus is further caused to:

use a backpropagation algorithm to compute gradients of the loss with respect to one or more parameters of the DSNN; and

apply an optimizer routine to compute the update.

3 . The apparatus of claim 1 , wherein the apparatus is further caused to:

encode one or more parts of a video in higher quality (HQ) content and remaining parts of the video in lower quality (LQ) content; and

use the HQ content to obtain the AGT.

4 . The apparatus of claim 1 , wherein the apparatus is further caused to lower a quality of the HQ content, and wherein to lower the quality of the HQ content the apparatus is further caused to perform at least one of the following:

encode the HQ content by using an encoding configuration that causes a decoded content to be of lower quality with respect to the HQ content and decode the encoded HQ content;

downsample the HQ content to a lower resolution; or

input the HQ content to a NN, and treat the output of the NN as a lower quality (LQ) content, wherein the NN is trained to output a content that has similar quality as the LQ content.

5 . The apparatus of claim 3 , wherein the apparatus is further caused to obtain the FDC at a decoder side by having an encoder encode the FDC as a low quality version of the HQ content, or as LQ content which is related or similar to the HQ content.

6 . The apparatus of claim 5 , wherein the apparatus is further caused to:

encode the LQ and HQ contents; and

compute the AGT as an output of a loss proxy neural network when an input to the loss proxy neural network comprises the HQ content or a content derived from the HQ content.

7 . The apparatus of claim 5 , wherein the apparatus is further caused to:

encode the LQ and HQ version of an input content; and

compute the AGT as an output of a loss proxy neural network when an input to the loss proxy neural network comprises the HQ content or a content derived from the HQ content.

8 . The apparatus of claim 5 , wherein the apparatus is further caused to:

extract a high resolution patch from the HQ content; and

extract a low resolution patch from the LQ content.

9 . The apparatus of claim 1 , wherein the apparatus is caused to:

store the finetuned DSNN in a buffer; and

signal the stored DSNN when the stored DSNN is to be used to process subsequent regions or frames or when the stored DSNN needs to be updated.

10 . A method comprising:

providing a finetuning driving content (FDC) or a content derived from FDC to a decoder side neural network (DSNN);

computing an output of the DSNN as a processed FDC;

computing a loss based on the processed FDC and an approximated ground truth data (AGT) associated with the FDC;

computing an update to the DSNN;

applying the computed update to the DSNN;

running a loss proxy neural network for one or more input frames; and

encoding an output of the loss proxy neural network, representing the AGT, as metadata.

11 . The method of claim 10 , wherein computing the update to the DSNN comprises:

using a backpropagation algorithm to compute gradients of the loss with respect to one or more parameters of the DSNN; and

applying an optimizer routine to compute the update.

12 . The method claim 10 , further comprising:

encoding one or more parts of a video in a higher quality (HQ) content and remaining parts of the video in a lower quality (LQ) content; and

using the HQ content to obtain the AGT.

13 . The method of claim 10 , further comprising lowering a quality of the HQ content, wherein lowering the quality of the HQ content comprises:

encoding the HQ content by using an encoding configuration that causes a decoded content to be of lower quality with respect to the HQ content and decode the encoded HQ content;

downsampling the HQ content to a lower resolution; or

inputting the HQ content to a NN, and treat the output of the NN as a lower quality (LQ) content, wherein the NN is trained to output a content that has similar quality as the LQ content.

14 . The method of claim 13 , further comprising obtaining the FDC at a decoder side by having an encoder encode the FDC as a low quality version of the HQ content, or as LQ content which is related or similar to the HQ content.

15 . The method of claim 14 , further comprising:

encoding the LQ and HQ contents; and

computing the AGT as an output of a loss proxy neural network when an input to the loss proxy neural network comprises the HQ content or a content derived from the HQ content.

16 . The method of claim 14 , further comprising:

encoding the LQ and HQ version of an input content; and

computing the AGT as an output of a loss proxy neural network when an input to the loss proxy neural network comprises the HQ content or a content derived from the HQ content.

17 . The method of claim 14 , further comprising:

extracting a high resolution patch from the HQ content; and

extracting a low resolution patch from the LQ content.

18 . The method of claim 10 , further comprising:

storing the finetuned DSNN in a buffer; and

signaling the stored DSNN when the stored DSNN is to be used to process subsequent regions or frames or when the stored DSNN needs to be updated.

Assignments (3)

Continuity (2)

References Cited (36)