End-to-end neural network based video coding

1. An apparatus for processing video data, comprising:

a memory; and

a processor coupled to the memory and configured to:

generate, using a first convolutional layer of an encoder sub-network of a neural network system, output values associated with a luminance channel of a frame, wherein the first convolutional layer of the encoder sub-network is configured to subsample the luminance channel of the frame to a resolution matching a resolution of at least one chrominance channel of the frame;

generate, using a second convolutional layer of the encoder sub-network, output values associated with the at least one chrominance channel of the frame;

generate a combined representation of the frame by combining the output values associated with the luminance channel of the frame and the output values associated with the at least one chrominance channel of the frame; and

generate encoded video data based on the combined representation of the frame.

2. The apparatus of claim 1 , wherein the processor is configured to:

process, using a first normalization layer of the encoder sub-network, the output values associated with the luminance channel of the frame; and

process, using a second normalization layer of the encoder sub-network, the output values associated with the at least one chrominance channel of the frame;

wherein the combined representation is generated based on an output of the first normalization layer and an output of the second normalization layer.

3. The apparatus of claim 2 , wherein the at least one chrominance channel and the luminance channel have different statistical characteristics, and wherein the first normalization layer and the second normalization layer are configured to balance statistical characteristics of the output values associated with the at least one chrominance channel and the output values associated with the luminance channel.

4. The apparatus of claim 1 , wherein a luminance filter of the first convolutional layer has a luminance filter size larger than a chrominance filter size of a chrominance filter of the second convolutional layer.

5. The apparatus of claim 1 , wherein the processor is further configured to:

generate the output values associated with the luminance channel of the frame based on input luminance values associated with the luminance channel of the frame, wherein the input luminance values are further associated with a luminance resolution; and

generate the output values associated with the at least one chrominance channel of the frame based on input chrominance values associated with the at least one chrominance channel of the frame, wherein the input chrominance values are further associated with a chrominance resolution, and wherein the luminance resolution is larger than the chrominance resolution.

6. The apparatus of claim 1 , wherein a luminance filter of the first convolutional layer has a luminance filter size five pixels by five pixels, and wherein a chrominance filter of the second convolutional layer has a chrominance filter size of three pixels by three pixels.

7. The apparatus of claim 1 , wherein the apparatus comprises a mobile device.

8. The apparatus of claim 1 , further comprising a display coupled to the processor.

9. A method of processing video data, the method comprising:

generating, by a first convolutional layer of an encoder sub-network of a neural network system, output values associated with a luminance channel of a frame, wherein the first convolutional layer of the encoder sub-network is configured to subsample the luminance channel of the frame to a resolution matching a resolution of at least one chrominance channel of the frame;

generating, by a second convolutional layer of the encoder sub-network, output values associated with the at least one chrominance channel of the frame;

generating a combined representation of the frame by combining the output values associated with the luminance channel of the frame and the output values associated with the at least one chrominance channel of the frame; and

generating encoded video data based on the combined representation of the frame.

10. The method of claim 9 , further comprising:

processing, using a first normalization layer of the encoder sub-network, the output values associated with the luminance channel of the frame; and

processing, using a second normalization layer of the encoder sub-network, the output values associated with at the at least one chrominance channel of the frame;

wherein the combined representation is generated based on an output of the first normalization layer and an output of the second normalization layer.

11. The method of claim 10 , further comprising:

obtaining an encoded frame generated from the at least one chrominance channel and the luminance channel;

generating, by a first convolutional layer of a decoder sub-network of the neural network system, reconstructed output values associated with a luminance channel of the encoded frame; and

generating, by a second convolutional layer of the decoder sub-network, reconstructed output values associated with at least one chrominance channel of the encoded frame.

12. The method of claim 10 , wherein the frame has a luminance-chrominance (YUV) format.

13. The method of claim 10 , wherein the at least one chrominance channel and the luminance channel have different statistical characteristics, and wherein the first normalization layer and the second normalization layer are configured to balance statistical characteristics of the output values associated with the at least one chrominance channel and the output values associated with the luminance channel.

14. The method of claim 9 , wherein a luminance filter of the first convolutional layer has a luminance filter size larger than a chrominance filter size of a chrominance filter of the second convolutional layer.

15. The method of claim 9 , further comprising:

generating the output values associated with the luminance channel of the frame based on input luminance values associated with the luminance channel of the frame, wherein the input luminance values are further associated with a luminance resolution; and

generating the output values associated with the at least one chrominance channel of the frame based on input chrominance values associated with the at least one chrominance channel of the frame, wherein the input chrominance values are further associated with a chrominance resolution, and wherein the luminance resolution is larger than the chrominance resolution.

16. An apparatus for processing video data, comprising:

a memory; and

a processor coupled to the memory and configured to:

obtain an encoded frame;

generate, using a first convolutional layer of a decoder sub-network of a neural network system, reconstructed output values associated with a luminance channel of the encoded frame, wherein the first convolutional layer of the decoder sub-network is configured to upsample the luminance channel to a resolution matching a resolution of at least one chrominance channel of the encoded frame;

generate, using a second convolutional layer of the decoder sub-network, reconstructed output values associated with the at least one chrominance channel of the encoded frame; and

generate an output frame including the reconstructed output values associated with the luminance channel and the reconstructed output values associated with the at least one chrominance channel.

17. The apparatus of claim 16 , wherein the processor is configured to:

process, using a first normalization layer of the decoder sub-network, values associated with the luminance channel of the encoded frame, wherein the reconstructed output values associated with the luminance channel are generated based on an output of the first normalization layer; and

process, using a second normalization layer of the decoder sub-network, values associated with the at least one chrominance channel of the encoded frame, wherein the reconstructed output values associated with the at least one chrominance channel are generated based on an output of the second normalization layer.

18. The apparatus of claim 16 , wherein a luminance filter of the first convolutional layer has a luminance filter size larger than a chrominance filter size of a chrominance filter of the second convolutional layer.

19. The apparatus of claim 18 , wherein the luminance filter size is nine pixels by nine pixels, and wherein the chrominance filter size is five pixels by five pixels.

20. The apparatus of claim 16 , wherein the processor is configured to:

entropy decode samples of the encoded frame.

21. The apparatus of claim 16 , wherein the processor is configured to:

generate, using a first convolutional layer of an encoder sub-network of the neural network system, output values associated with the luminance channel of a frame;

generate, using a second convolutional layer of the encoder sub-network, output values associated with the at least one chrominance channel of the frame;

generate a combined representation of the frame by combining the output values associated with the luminance channel of the frame and the output values associated with the at least one chrominance channel of the frame; and

generate the encoded frame based on the combined representation of the frame.

22. The apparatus of claim 16 , wherein the processor includes a neural processing unit (NPU).

23. The apparatus of claim 16 , further comprising a display coupled to the processor, wherein the output frame comprises a video frame output to the display of the apparatus.

24. A method of processing video data, the method comprising:

obtaining an encoded frame;

generating, by a first convolutional layer of a decoder sub-network of a neural network system, reconstructed output values associated with a luminance channel of the encoded frame, wherein the first convolutional layer of the decoder sub-network is configured to upsample the luminance channel to a resolution matching a resolution of at least one chrominance channel of the encoded frame;

generating, by a second convolutional layer of the decoder sub-network, reconstructed output values associated with the at least one chrominance channel of the encoded frame; and

generating an output frame including the reconstructed output values associated with the luminance channel and the reconstructed output values associated with the at least one chrominance channel.

25. The method of claim 24 , further comprising:

processing, using a first normalization layer of the decoder sub-network, values associated with the luminance channel of the encoded frame, wherein the reconstructed output values associated with the luminance channel are generated based on an output of the first normalization layer; and

processing, using a second normalization layer of the decoder sub-network, values associated with the at least one chrominance channel of the encoded frame, wherein the reconstructed output values associated with the at least one chrominance channel are generated based on an output of the second normalization layer.

26. The method of claim 24 , wherein a luminance filter of the first convolutional layer has a luminance filter size larger than a chrominance filter size of a chrominance filter of the second convolutional layer of the decoder sub-network.

27. The method of claim 26 , wherein the luminance filter size is nine pixels by nine pixels, and wherein the chrominance filter size is five pixels by five pixels.

28. The method of claim 24 , further comprising entropy decoding samples of the encoded frame.

29. The method of claim 24 , further comprising:

generating, using a first convolutional layer of an encoder sub-network of the neural network system, output values associated with the luminance channel of a frame;

generating, using a second convolutional layer of the encoder sub-network, output values associated with the at least one chrominance channel of the frame;

generating a combined representation of the frame by combining the output values associated with the luminance channel of the frame and the output values associated with the at least one chrominance channel of the frame; and

generating the encoded frame based on the combined representation of the frame.

30. The method of claim 23 , wherein the neural network system is part of a neural processing unit (NPU).