Video coding with neural network based in-loop filtering
An electronic apparatus performs a method of decoding video data, including: reconstructing, from a video bitstream, a picture frame that includes a luma component, a first and a second chroma components, and applying a trained neural network based in-loop filter to the reconstructed picture frame by: converting a first resolution of the samples of the at least one of the first and the second chroma components to a second resolution of the samples of the luma component when the first resolution is different from the second resolution; concatenating samples of at least one of the first and the second chroma components with the luma component; processing the concatenated samples using a convolutional neural network; and reconverting the samples of the at least one of the first and the second chroma components processed by the convolutional neural network from the second resolution back to the first resolution.
1. A method of decoding video data, comprising:
reconstructing, from a video bitstream, a picture frame that includes a luma component, a first chroma component, and a second chroma component, and
applying a trained neural network based in-loop filter to the reconstructed picture frame by performing operations comprising:
concatenating samples of at least one of the first and the second chroma components with the luma component to create concatenated samples; and
processing the concatenated samples using a convolutional neural network
wherein the trained neural network based in-loop filter has been trained to obtain model parameters by reducing difference between output reconstructed picture frames from the convolutional neural network and corresponding ground-truth picture frames,
wherein applying the trained neural network based in-loop filter to the reconstructed picture frame further comprises:
converting a first resolution of the samples of the at least one of the first and the second chroma components to a second resolution of the samples of the luma component when the first resolution of the at least one of the first and the second chroma components is different from the second resolution of the luma component; and
reconverting the samples of the at least one of the first and the second chroma components processed by the convolutional neural network from the second resolution back to the first resolution.
2. The method according to claim 1 , wherein the converting the first resolution of the samples of the at least one of the first and the second chroma components to a second resolution of the samples of the luma component, comprises:
up-sampling the samples of at least one of the first and the second chroma components from the first resolution to the second resolution of the samples of the luma component, and
wherein the reconverting the samples of the at least one of the first and the second chroma components processed by the convolutional neural network from the second resolution back to the first resolution comprises:
down-sampling the samples of the at least one of the first and the second chroma components processed by the convolutional neural network from the second resolution back to the first resolution.
3. The method according to claim 1 , wherein applying the trained neural network based in-loop filter to the reconstructed picture frame further comprises:
feeding quantization parameter (QP) information of prediction residuals as an additional input to the convolutional neural network.
4. The method according to claim 3 , wherein the QP information is normalized QP value, or
the QP information is QP step size, or
the QP information is QP maps, each of the QP maps combined with a corresponding color plane of the luma component, the first chroma component, and the second chroma component, respectively.
5. The method according to claim 1 , wherein the corresponding ground-truth picture frames are available at an encoder.
6. The method according to claim 5 , wherein the model parameters are explicitly signaled from the encoder to a decoder.
7. The method according to claim 1 , wherein the trained neural network based in-loop filter has been trained for each encoded picture frame and updated frequently.
8. The method according to claim 1 , wherein the trained neural network based in-loop filter has been trained for each encoded picture frame and updated frequently with overfitting enabled.
9. The method according to claim 1 , wherein the trained neural network based in-loop filter has been trained and saved offline for next use.
10. The method according to claim 1 , wherein the trained neural network based in-loop filter is-has been trained and saved offline for next use with overfitting disabled.
11. The method according to claim 1 , wherein the converting and the reconverting are processed by convolution modules.
12. The method according to claim 1 , wherein the trained neural network based in-loop filter has been trained from reconstructed picture frame training data with a deblocking filter applied.
13. The method according to claim 1 , wherein the trained neural network based in-loop filter has been trained from reconstructed picture frame training data without a deblocking filter applied.
14. The method according to claim 1 , wherein applying the trained neural network based in-loop filter to the reconstructed picture frame further comprises: segmenting the reconstructed picture frame into a plurality of overlapping blocks, wherein an overlapping size of the overlapping blocks is selected according to one or more of a content of the reconstructed picture frame, width to height ratio of the overlapping blocks, and width to height ratio of the reconstructed picture frame.
15. The method according to claim 1 , wherein applying the trained neural network based in-loop filter to the reconstructed picture frame further comprises: rounding sample values at an output of the neural network by applying an offset value, wherein the offset value is a fixed value or a value signaled and determined from an encoder.
16. The method according to claim 1 , wherein the convolutional neural network is ResNet.
17. An electronic apparatus comprising:
one or more processing units;
memory coupled to the one or more processing units; and
a plurality of programs stored in the memory that, when executed by the one or more processing units, cause the electronic apparatus to perform decoding video data by performing operations comprising:
reconstructing, from a video bitstream, a picture frame that includes a luma component, a first chroma component, and a second chroma component, and
applying a trained neural network based in-loop filter to the reconstructed picture frame by performing operations comprising:
concatenating samples of at least one of the first and the second chroma components with the luma component to create concatenated samples; and
processing the concatenated samples using a convolutional neural network,
wherein the trained neural network based in-loop filter has been trained to obtain model parameters by reducing difference between output reconstructed picture frames from the convolutional neural network and corresponding ground-truth picture frames,
wherein applying the trained neural network based in-loop filter to the reconstructed picture frame further comprises:
converting a first resolution of the samples of the at least one of the first and the second chroma components to a second resolution of the samples of the luma component when the first resolution of the at least one of the first and the second chroma components is different from the second resolution of the luma component; and
reconverting the samples of the at least one of the first and the second chroma components processed by the convolutional neural network from the second resolution back to the first resolution.
18. A non-transitory computer readable storage medium storing a plurality of programs for execution by an electronic apparatus having one or more processing units, wherein the plurality of programs, when executed by the one or more processing units, cause the electronic apparatus to perform decoding video data comprising by performing operations comprising:
reconstructing, from a video bitstream, a picture frame that includes a luma component, a first chroma component, and a second chroma component, and
applying a trained neural network based in-loop filter to the reconstructed picture frame by performing operations comprising:
concatenating samples of at least one of the first and the second chroma components with the luma component to create concatenated samples; and
processing the concatenated samples using a convolutional neural network
wherein the trained neural network based in-loop filter has been trained to obtain model parameters by reducing difference between output reconstructed picture frames from the convolutional neural network and corresponding ground-truth picture frames,
wherein applying the trained neural network based in-loop filter to the reconstructed picture frame further comprises:
converting a first resolution of the samples of the at least one of the first and the second chroma components to a second resolution of the samples of the luma component when the first resolution of the at least one of the first and the second chroma components is different from the second resolution of the luma component; and
reconverting the samples of the at least one of the first and the second chroma components processed by the convolutional neural network from the second resolution back to the first resolution.