Scene-adaptive online learning for video post processing

1 . A method of processing video data at a capture device, the method comprising:

encoding a set of original pictures to create encoded video data;

decoding the encoded video data to create a set of reconstructed pictures;

determining a subset of parameters to update from among a plurality of parameters of a post-processing filter network, including using the set of original pictures as ground truth and the set of reconstructed pictures as input to the post-processing filter network, wherein determining the subset of parameters to update comprises:

accumulating a respective gradient change for every parameter of the plurality of parameters over N forward and backward passes;

sorting the respective gradient changes in a list; and

selecting parameters associated with a top ‘n’ percent of the respective gradient changes in the list as the subset of parameters;

updating the subset of parameters to generate updated parameters; and

sending the encoded video data and the updated parameters to a playback device.

2 . The method of claim 1 , wherein determining the subset of parameters to update comprises:

accumulating a respective gradient change for every parameter of the plurality of parameters over N forward and backward passes;

determining a respective accumulated gradient change ratio for every parameter of the plurality of parameters, wherein the respective accumulated gradient change ratio is a function of the respective gradient change divided by a magnitude of a respective parameter;

sorting the respective accumulated gradient change ratios in a list; and

selecting parameters associated with a top ‘n’ percent of the respective accumulated gradient change ratios in the list as the subset of parameters.

3 . The method of claim 1 , wherein determining the subset of parameters to update comprises one of:

randomly selecting ‘n’ percent of the plurality of parameters as the subset of parameters; or

selecting ‘n’ percent of contiguous parameters starting from a last layer of the post-processing filter network.

4 . The method of claim 1 , wherein updating the subset of parameters to generate updated parameters comprises:

performing one or more of a forward gradient method, back propagation, or a mixture of forward and backward training to update the subset of parameters.

5 . The method of claim 4 , further comprising:

not updating values of the plurality of parameters that are not in the subset of parameters.

6 . The method of claim 1 , wherein the set of original pictures is one of a first number pictures in a sequence, or a uniformly selected first number of pictures in a sequence.

7 . The method of claim 1 , further comprising:

selecting one or more pictures for the set of original pictures based on a change in signal-to-noise ratio between pictures.

8 . The method of claim 1 , further comprising:

selecting one or more pictures for the set of original pictures based on a scene change.

9 . The method of claim 1 , further comprising:

compressing the updated parameters.

10 . The method of claim 9 , wherein compressing the updated parameters comprises:

determining an arithmetic parameter ‘n’ based on a percentage of updated parameters to a total number of the plurality of parameters of the post-processing filter network, wherein arithmetic parameter ‘n’ is a probability used in arithmetic coding of the updated parameters;

arithmetically encoding a mask using arithmetic parameter ‘n’, wherein each value of the mask indicates whether or not a parameter in the post-processing filter network is the updated parameter;

determining a respective parameter difference between a respective updated parameter and a corresponding parameter of the subset of parameters; and

encoding the respective parameters differences for the updated parameters.

11 . The method of claim 1 , wherein the post-processing filter network is a convolutional neural network.

12 . An apparatus configured to process video data at a capture device, the apparatus comprising:

a memory; and

processing circuitry in communication with the memory, the processing circuitry configured to:

encode a set of original pictures to create encoded video data;

decode the encoded video data to create a set of reconstructed pictures;

determine a subset of parameters to update from among a plurality of parameters of a post-processing filter network, including using the set of original pictures as ground truth and the set of reconstructed pictures as input to the post-processing filter network, wherein to determine the subset of parameters to update, the processing circuitry is further configured to:

accumulate a respective gradient change for every parameter of the plurality of parameters over N forward and backward passes;

sort the respective gradient changes in a list; and

select parameters associated with a top ‘n’ percent of the respective gradient changes in the list as the subset of parameters;

update the subset of parameters to generate updated parameters; and

send the encoded video data and the updated parameters to a playback device.

13 . The apparatus of claim 12 , wherein to determine the subset of parameters to update, the processing circuitry is further configured to:

accumulate a respective gradient change for every parameter of the plurality of parameters over N forward and backward passes;

determine a respective accumulated gradient change ratio for every parameter of the plurality of parameters, wherein the respective accumulated gradient change ratio is a function of the respective gradient change divided by a magnitude of a respective parameter;

sort the respective accumulated gradient change ratios in a list; and

select parameters associated with a top ‘n’ percent of the respective accumulated gradient change ratios in the list as the subset of parameters.

14 . The apparatus of claim 12 , wherein to determine the subset of parameters to update, the processing circuitry is further configured to:

randomly select ‘n’ percent of the plurality of parameters as the subset of parameters; or

select ‘n’ percent of contiguous parameters starting from a last layer of the post-processing filter network.

15 . The apparatus of claim 12 , wherein to update the subset of parameters to generate updated parameters, the processing circuitry is configured to:

perform one or more of a forward gradient method, back propagation, or a mixture of forward and backward training to update the subset of parameters.

16 . The apparatus of claim 15 , wherein the processing circuitry is further configured to:

not update values of the plurality of parameters that are not in the subset of parameters.

17 . The apparatus of claim 12 , wherein the set of original pictures is one of a first number pictures in a sequence, or a uniformly selected first number of pictures in a sequence.

18 . The apparatus of claim 12 , wherein the processing circuitry is further configured to:

select one or more pictures for the set of original pictures based on a change in signal-to-noise ratio between pictures.

19 . The apparatus of claim 12 , wherein the processing circuitry is further configured to:

select one or more pictures for the set of original pictures based on a scene change.

20 . The apparatus of claim 12 , wherein the processing circuitry is further configured to:

compress the updated parameters.

21 . The apparatus of claim 20 , wherein to compress the updated parameters, the processing circuitry is configured to:

determine an arithmetic parameter ‘n’ based on a percentage of updated parameters to a total number of the plurality of parameters of the post-processing filter network, wherein arithmetic parameter ‘n’ is a probability used in arithmetic coding of the updated parameters;

arithmetically encode a mask using arithmetic parameter ‘n’, wherein each value of the mask indicates whether or not a parameter in the post-processing filter network is the updated parameter;

determine a respective parameter difference between a respective updated parameter and a corresponding parameter of the subset of parameters; and

encode the respective parameters differences for the updated parameters.

22 . The apparatus of claim 12 , wherein the post-processing filter network is a convolutional neural network.

23 . The apparatus of claim 12 , further comprising:

a camera configured to capture the set of original pictures.

24 . A method of processing video data at a playback device, the method comprising:

receiving encoded video data and encoded parameters for a post-processing filter network, wherein the encoded parameters include an arithmetic parameter ‘n’, an arithmetically encoded mask, and a respective parameter difference for each of the updated parameters;

decoding the encoded video data to generate a decoded picture;

decoding the encoded parameters to recover updated parameters, wherein decoding the encoded parameters comprises:

decoding the arithmetic parameter ‘n’;

arithmetically decoding the arithmetically encoded mask using arithmetic parameter ‘n’ to recover a mask, wherein each value of the mask indicates whether or not a parameter in the post-processing filter network is the updated parameter; and

adding the respective parameter difference to a corresponding parameter value of the post-processing filter network based on the mask to recover the updated parameters; and;

processing the decoded picture using the post-processing filter network and the updated parameters.

25 . An apparatus configured to process video data at a playback device, the apparatus comprising:

a memory; and

processing circuitry in communication with the memory, the processing circuitry configured to:

receive encoded video data and encoded parameters for a post-processing filter network, wherein the encoded parameters include an arithmetic parameter ‘n’, an arithmetically encoded mask, and a respective parameter difference for each of the updated parameters;

decode the encoded video data to generate a decoded picture;

decode the encoded parameters to recover updated parameters, wherein to decode the encoded parameters, the processing circuitry is further configured to:

decode the arithmetic parameter ‘n’; arithmetically decode the arithmetically encoded mask using arithmetic parameter ‘n’ to recover a mask, wherein each value of the mask indicates whether or not a parameter in the post-processing filter network is the updated parameter; and

add the respective parameter difference to a corresponding parameter value of the post-processing filter network based on the mask to recover the updated parameters; and

process the decoded picture using the post-processing filter network and the updated parameters to generate a filtered picture.

26 . The apparatus of claim 25 , further comprising:

a display configured to display the filtered picture.