Training end-to-end video processes
Methods and systems for optimising the quality of visual data. Specifically, methods and systems for preserving visual information during compression and decompression. An example method for optimising visual data includes using a pre-processing neural network to optimise visual data prior to encoding the visual data in visual data processing; and using a post-processing neural network to enhance visual data following decoding visual data in visual data processing.
1. A method for optimising visual data, the method comprising:
using a pre-processing neural network on input pixel-based data to produce pre-processed pixel-based visual data for an encoding process;
encoding the pre-processed pixel-based visual data using an encoder to produce encoded pixel-based visual data;
decoding the encoded pixel-based visual data using a decoder to produce decoded pixel-based visual data; and
using a post-processing neural network on the decoded pixel-based visual data to produce enhanced pixel-based visual data,
wherein the pre-processing neural network and the post-processing neural network have been jointly trained to improve the enhanced pixel-based visual data.
2. The method of claim 1 wherein the pre-processing neural network and the post-processing neural network are jointly trained for a codec used by the encoder and the decoder.
3. The method of claim 2 , wherein the pre-processing neural network and the post-processing neural network are jointly trained using a differential approximation based on the codec.
4. The method of claim 1 , wherein the pre-processing neural network and the post-processing neural network are trained for specific source material.
5. The method of claim 1 where the pre-processing neural network and the post-processing neural network are selected from a library of network pairs, each pair of networks in being trained for specific source material.
6. The method of claim 1 , wherein the pre-processing neural network and the post-processing neural network are selected from a library of neural networks based on a similarity between features associated with the pre-processing neural network and features of the input pixel-based data.
7. The method of claim 1 , wherein the pre-processing neural network and the post-processing neural network are selected from a library of neural networks based on artefact severity of the decoded pixel-based visual data.
8. The method of claim 7 , wherein the pre-processing neural network and the post-processing neural network are selected from a library of neural networks based on a classification of the pre-processing neural network and the post-processing neural network.
9. The method of claim 8 , wherein the classification indicates a specific scene and artefact severity.
10. The method of claim 1 , wherein the pre-processing neural network and the post-processing neural network are jointly trained for a specific scene and artefact severity.
11. A non-transitory computer program product comprising instructions that, when executed by a processor, cause a computing device to perform operations including:
using a pre-processing neural network on input pixel-based data to produce pre-processed pixel-based visual data for an encoding process;
encoding the pre-processed pixel-based visual data using an encoder to produce encoded pixel-based visual data;
decoding the encoded pixel-based visual data using a decoder to produce decoded pixel-based visual data; and
using a post-processing neural network on the decoded pixel-based visual data to produce enhanced pixel-based visual data,
wherein the pre-processing neural network and the post-processing neural network have been jointly trained to improve the enhanced pixel-based visual data.
12. The computer program product of claim 11 wherein the pre-processing neural network and the post-processing neural network are jointly trained for a codec used by the encoder and the decoder.
13. The computer program product of claim 12 , wherein the pre-processing neural network and the post-processing neural network are jointly trained using a differential approximation based on the codec.
14. The computer program product of claim 11 , wherein the pre-processing neural network and the post-processing neural network are trained for specific source material.
15. The computer program product of claim 11 where the pre-processing neural network and the post-processing neural network are selected from a library of network pairs, each pair of networks in being trained for specific source material.
16. The computer program product of claim 11 , wherein the pre-processing neural network and the post-processing neural network are selected from a library of neural networks based on a similarity between features associated with the pre-processing neural network and features of the input pixel-based data.
17. The computer program product of claim 11 , wherein the pre-processing neural network and the post-processing neural network are selected from a library of neural networks based on artefact severity of the decoded pixel-based visual data.
18. The computer program product of claim 17 , wherein the pre-processing neural network and the post-processing neural network are selected from a library of neural networks based on a classification of the pre-processing neural network and the post-processing neural network.
19. The computer program product of claim 11 , wherein the pre-processing neural network and the post-processing neural network are jointly trained for a specific scene and artefact severity.
20. A computing system comprising:
a first computing device including at least one processor and memory storing instructions that, when executed by the at least one processor, cause the first computing device to perform operations including:
using a pre-processing neural network on input pixel-based data to produce pre-processed pixel-based visual data for an encoding process,
encoding the pre-processed pixel-based visual data using an encoder to produce encoded pixel-based visual data, and
transmitting the encoded pixel-based visual data to a second computing device; and
the second computing device including at least one processor and memory storing instructions that, when executed by the at least one processor, cause the second computing device to perform operations including:
decoding the encoded pixel-based visual data using a decoder to produce decoded pixel-based visual data, and
using a post-processing neural network on the decoded pixel-based visual data to produce enhanced pixel-based visual data,
wherein the pre-processing neural network and the post-processing neural network have been jointly trained to improve the enhanced pixel-based visual data.