Training end-to-end video processes
Disclosed is method for training a plurality of visual processing algorithms for processing visual data. The method includes using a pre-processing hierarchical algorithm to process the visual data prior to encoding the visual data in visual data processing, and using a post-processing hierarchical algorithm to further process the visual data following decoding visual data in visual data processing. The encoding and decoding are performed with respect to a predetermined visual data codec and may be content specific.
1. A method for jointly training a pre-processing neural network and a post-processing neural network for a visual data encoding and decoding process, the method comprising:
determining a differential approximation of the encoding and decoding process;
receiving one or more sections of input visual data;
using a pre-processing neural network to process the input visual data and to output pre-processed visual data;
applying the encoding and decoding process to the pre-processed visual data to generate decoded visual data;
using a post-processing neural network to further process the decoded visual data and to output reconstructed visual data;
comparing the reconstructed visual data with the input visual data using a metric; and
updating parameters of the pre-processing neural network and the post-processing neural network using the differential approximation of the encoding and decoding process and based on the comparing of the reconstructed visual data with the input visual data.
2. The method according to claim 1 , wherein updating the parameters of the pre-processing neural network and the post-processing neural network includes optimising the pre-processing neural network and the post-processing neural network with respect to the metric.
3. The method according to claim 1 , wherein one or more parameters associated with the pre-processing neural network is stored in a library for re-use in encoding alternative visual data similar to the visual data used for training.
4. The method according to claim 1 , further comprising:
transmitting the one or more updated parameters associated with the pre-processing neural network to a device configured to use the pre-processing neural network with the updated parameters in encoding visual data similar to the input visual data.
5. The method according to claim 1 , further comprising:
transmitting the one or more updated parameters with the post-processing neural network with processed visual data to a remote device configured to use the post-processing neural network with the updated parameters, wherein the remote device has another pre-processing neural network associated with the post-processing neural network.
6. The method according to claim 1 , wherein at least one of:
the pre-processing neural network includes a layer that generalises the visual data processing, and
the post-processing neural network includes a layer that generalises the visual data processing.
7. The method according to claim 1 , wherein receiving a plurality of predetermined bit rate for use in training the pre-processing neural network or in training the post-processing neural network.
8. The method according to claim 1 , wherein input visual data includes at least one of:
a single frame of visual data,
a sequence of frames of visual data, and
a region within a frame or sequence of frames of visual data.
9. The method according to claim 1 , wherein the pre-processing neural network is different for each section of the visual data.
10. The method according to claim 1 , wherein the post-processing neural network is developed using a machine learning approach.
11. The method according to claim 1 , wherein the post-processing neural network is a non-linear neural network including at least one convolutional neural networks.
12. The method according to claim 1 , wherein one of:
the pre-processing neural network is used as a first filter when encoding the visual data, and
the post-processing neural network is used as a second filter when decoding the visual data.
13. The method according to claim 1 , wherein the pre-processing neural network uses a spatio-temporal approach.
14. The method according to claim 1 , wherein using the pre-processing or post-processing neural network includes at least one of:
training the neural network,
generating the neural network, and
developing the neural network.
15. The method according to claim 2 , wherein the pre-processing neural network and the post-processing neural network are optimized based on a tradeoff between compression and reconstruction error.
16. The method according to claim 1 , wherein the differential approximation of the encoding and decoding process includes neural network layer.
17. The method according to claim 1 , wherein the updating the parameters of the pre-processing neural network and the post-processing neural network include applying a backpropagation algorithm using gradients of the differential approximation to the encoding and decoding process.