Methods and apparatus for hybrid training of neural networks for video coding
A method and an apparatus for hybrid training of neural networks for video coding are provided. The method includes: obtaining, in an offline training stage, an offline trained network by training a first neural network offline using a plurality of first data sets; refining, in an online training stage, a plurality of neural network layers using a plurality of second data sets, wherein the plurality of neural network layers comprise at least one neural network layer in the offline trained network or in a second neural network connected to the offline trained network.
1 . A method for hybrid training of neural networks for video coding, comprising:
obtaining, in an offline training stage, an offline trained network by training a first neural network offline using a plurality of first data sets; and
refining, in an online training stage, a plurality of neural network layers using a plurality of second data sets, wherein the plurality of neural network layers comprise at least one neural network layer in the offline trained network or in a second neural network connected to the offline trained network,
wherein the plurality of second data sets comprise coding information, video content, and/or information other than video frames,
the coding information comprises temporal layer information,
the video content comprises a plurality of patches extracted based on regions of interest from a video frame, and
the information other than video frames comprises motion information.
2 . The method of claim 1 , wherein refining the plurality of neural network layers using the plurality of second data sets comprises repeatedly refining the plurality of neural network layers using the plurality of second data sets.
3 . The method of claim 1 , wherein the offline trained network comprises more neural network layers than the plurality of neural network layers that are refined.
4 . The method of claim 1 , wherein the offline trained network comprises a plurality of residual blocks, the plurality of neural network layers that are refined comprise a last residual block in the offline trained network.
5 . The method of claim 1 , wherein a number of neural network layers of the second neural network is less than a number of neural network layers of the offline trained network.
6 . The method of claim 1 , wherein the second neural network comprises at least one convolutional layers and, and the plurality of neural network layers that are refined comprise the at least one convolutional layers in the second neural network.
7 . The method of claim 1 , further comprising:
performing different learning algorithms in a same training stage or different training stages, wherein the training stage is the offline training stage or the online training stage; or
applying different starting values of a learning rate parameter in a learning algorithm in a same training stage or different training stages.
8 . The method of claim 7 , wherein performing different learning algorithms in different training stages comprises:
performing a stochastic gradient descent (SGD) algorithm in the offline training stage and performing an adaptive moment estimation (ADAM) algorithm in the online training stage.
9 . The method of claim 7 , wherein performing different learning algorithms in a same training stage comprises:
in response to determining that a loss generated by a loss function is less than a predefined threshold value, switching from an adaptive moment estimation (ADAM) algorithm to a stochastic gradient descent (SGD) algorithm in the offline training stage.
10 . The method of claim 1 , wherein the plurality of neural network layers are refined based on at least one of following manners: periodically or in response to a triggering event.
11 . The method of claim 10 , wherein the triggering event comprises one of following events:
a video encoder behavior;
a change of a video encoder configuration; or
a change of video content, wherein the video content comprises an object or background of an object within a video frame.
12 . The method of claim 1 , wherein the plurality of patches are selected from patches belonging to a same cluster or patches belonging to different clusters, wherein different patches that are cropped from a video frame correspond to different clusters based on representation similarities or levels of distortion.
13 . The method of claim 1 , wherein a data amount of the plurality of first data sets is more than a data amount of the plurality of second data sets.
14 . The method of claim 2 , wherein repeatedly refining the plurality of neural network layers comprises at least one of following steps:
controlling a trainable structure of network parameters by configuring a sub-set of the network parameters to be trainable;
controlling a dynamical range of network parameters by controlling data types of the network parameters; or
controlling network parameters of the plurality of neural network layers using weight regularization.
15 . The method of claim 1 , further comprising:
obtaining a plurality of refined network parameters by repeatedly refining the plurality of neural network layers;
obtaining a plurality of processed network parameters based on the plurality of refined network parameters by one of following operations:
pruning or quantizing the plurality of refined network parameters; or
processing the plurality of refined network parameters using mean-shift or weight sharing; and
signaling the plurality of processed network parameters.
16 . The method of claim 1 , further comprising:
obtaining a plurality of refined network parameters by repeatedly refining the plurality of neural network layers;
obtaining a difference between original network parameters of the plurality of neural network layers and the plurality of refined network parameters; and
constructing a differential map based on the difference.
17 . The method of claim 1 , further comprising:
obtaining a plurality of refined network parameters by repeatedly refining the plurality of neural network layers; and
extending or tailoring filter coefficients of an adaptive in-loop filter (ALF) in an encoder to approximate the plurality of refined network parameters.
18 . An apparatus for hybrid training of neural networks for video coding, comprising:
one or more processors; and
a memory configured to store instructions executable by the one or more processors,
wherein the one or more processors, upon execution of the instructions, are configured to perform:
obtaining, in an offline training stage, an offline trained network by training a first neural network offline using a plurality of first data sets; and
refining, in an online training stage, a plurality of neural network layers using a plurality of second data sets, wherein the plurality of neural network layers comprise at least one neural network layer in the offline trained network or in a second neural network connected to the offline trained network,
wherein the plurality of second data sets comprise coding information, video content, and/or information other than video frames,
the coding information comprises temporal layer information,
the video content comprises a plurality of patches extracted based on regions of interest from a video frame, and
the information other than video frames comprises motion information.
19 . A non-transitory computer-readable storage medium storing a bitstream formed by instructions which when executed by a computing device having one or more processors, cause the one or more processors to perform operations comprising:
obtaining, in an offline training stage, an offline trained network by training a first neural network offline using a plurality of first data sets; and
refining, in an online training stage, a plurality of neural network layers using a plurality of second data sets, wherein the plurality of neural network layers comprise at least one neural network layer in the offline trained network or in a second neural network connected to the offline trained network,
wherein the plurality of second data sets comprise coding information, video content, and/or information other than video frames,
the coding information comprises temporal layer information,
the video content comprises a plurality of patches extracted based on regions of interest from a video frame, and
the information other than video frames comprises motion information.