Processing image data
View Patent ↗A method of processing, prior to encoding using an external encoder, image data using an artificial neural network is provided. The external encoder is operable in a plurality of encoding modes. At the neural network, image data representing one or more images is received. The image data is processed using the neural network to generate output data indicative of an encoding mode selected from the plurality of encoding modes of the external encoder. The neural network trained to select using image data an encoding mode of the plurality of encoding modes of the external encoder using one or more differentiable functions configured to emulate an encoding process. The generated output data is outputted from the neural network to the external encoder to enable the external encoder to encode the image data using the selected encoding mode.
1. A computer-implemented method of processing, prior to encoding using an external encoder, image data using an artificial neural network, wherein the external encoder is operable in a plurality of encoding modes, the method comprising:
receiving, at the artificial neural network, the image data representing one or more images;
processing the image data using the artificial neural network to generate output data indicative of a selected encoding mode selected from the plurality of encoding modes of the external encoder, wherein:
the artificial neural network is trained to select the selected encoding mode from the plurality of encoding modes using the image data and one or more differentiable functions configured to emulate an encoding process, the one or more differentiable functions being configured to emulate operations associated with the plurality of encoding modes of the external encoder, and
the artificial neural network is trained independently of the external encoder; and
outputting the generated output data from the artificial neural network to the external encoder to enable the external encoder to encode the image data using the selected encoding mode.
2. The method according to claim 1 , wherein the plurality of encoding modes comprise a plurality of prediction modes for encoding the image data using predictive coding, the plurality of prediction modes relating to intra-prediction and/or inter-prediction.
3. The method according to claim 1 , wherein one or more of the plurality of encoding modes comprise a plurality of quantization parameters useable by the external encoder to encode the image data.
4. The method according to claim 1 , wherein the plurality of encoding modes are associated with an image and/or video coding standard.
5. The method according to claim 1 , wherein each of the plurality of encoding modes generates an encoded bitstream having a format that is compliant with an image and/or video coding standard.
6. The method according to claim 1 , wherein the artificial neural network is configured to select the selected encoding mode from the plurality of encoding modes based on image content of the received image data.
7. The method according to claim 1 , wherein the artificial neural network is trained to optimize a rate score indicative of bits required by the external encoder to encode output pixel representations generated using the plurality of encoding modes.
8. The method according to claim 7 , wherein the rate score is calculated using the one or more differentiable functions configured to emulate the encoding process.
9. The method according to claim 7 , wherein the rate score is calculated using a differentiable rate loss function.
10. The method according to claim 7 , further comprising generating the output pixel representations at the artificial neural network.
11. The method according to claim 1 , wherein the artificial neural network is trained to optimize a quality score indicative of a quality of reconstructed pixel representations generated using the plurality of encoding modes.
12. The method according to claim 11 , wherein the quality score is calculated using the one or more differentiable functions configured to emulate the encoding process.
13. The method according to claim 11 , wherein the quality score is calculated using a differentiable quality loss function.
14. The method according to claim 11 , wherein the quality score is indicative of at least one of:
signal distortion in the reconstructed pixel representations; or
loss of perceptual and/or aesthetic quality in the reconstructed pixel representations.
15. The method according to claim 1 , wherein the artificial neural network is trained using one or more regularization coefficients corresponding to a desired rate-quality operational point.
16. The method according to claim 1 , further comprising:
determining one or more loss functions based on the generated output data; and
adjusting the artificial neural network using back-propagation of values of the one or more loss functions.
17. The method according to claim 1 , further comprising, at the external encoder:
receiving the output data from the artificial neural network; and
encoding the image data using the encoding mode selected to generate an encoded bitstream.
18. A computing device configured to perform a method of processing image data using an artificial neural network prior to encoding using an external encoder operable in a plurality of encoding modes, comprising:
one or more memories comprising computer-executable instructions; and
one or more processors configured to execute the computer-executable instructions and cause the computing device to:
receive, at the artificial neural network, the image data representing one or more images;
process the image data using the artificial neural network to generate output data indicative of a selected encoding mode selected from the plurality of encoding modes of the external encoder, wherein:
the artificial neural network is trained to select the selected encoding mode of the plurality of encoding modes using the image data and one or more differentiable functions configured to emulate an encoding process, the one or more differentiable functions being configured to emulate operations associated with the plurality of encoding modes of the external encoder, and
the artificial neural network is trained independently of the external encoder; and
output the generated output data from the artificial neural network to the external encoder to enable the external encoder to encode the image data using the selected encoding mode.
19. A non-transitory computer-readable medium comprising computer-executable instructions that, when executed by a processor of a computing device, cause the computing device to perform a method of processing, prior to encoding using an external encoder, image data using an artificial neural network, wherein the external encoder is operable in a plurality of encoding modes, the method comprising:
receiving, at the artificial neural network, the image data representing one or more images;
processing the image data using the artificial neural network to generate output data indicative of a selected encoding mode selected from the plurality of encoding modes of the external encoder, wherein:
the artificial neural network is trained to select the selected encoding mode from the plurality of encoding modes using the image data and one or more differentiable functions configured to emulate an encoding process, the one or more differentiable functions being configured to emulate operations associated with the plurality of encoding modes of the external encoder, and
the artificial neural network is trained independently of the external encoder; and
outputting the generated output data from the artificial neural network to the external encoder to enable the external encoder to encode the image data using the selected encoding mode.