Scalable coding of video and associated features
The present disclosure relates to scalable encoding and decoding of pictures. In particular, a picture is processed by one or more network layers of a trained module to obtain base layer features. Then, enhancement layer features are obtained, e.g. by a trained network processing in sample domain. The base layer features are for use in computer vision processing. The base layer features together with enhancement layer features are for use in picture reconstruction, e.g. for human vision. The base layer features and the enhancement layer features are coded in a respective base layer bitstream and an enhancement layer bitstream. Accordingly, a scalable coding is provided which supports computer vision processing and/or picture reconstruction.
1 . An apparatus for processing a bitstream, the apparatus comprising:
a memory comprising instructions and a processing circuitry configured to execute the instructions to cause the apparatus to:
obtain a base layer bitstream including base layer features of a latent space and an enhancement layer bitstream including enhancement layer features;
extract from the base layer bitstream the base layer features; and
perform:
computer vision processing based on the base layer features; and
extracting the enhancement layer features from the enhancement layer bitstream and reconstructing a picture based on the base layer features and the enhancement layer features, and the reconstructing of the picture comprises:
combining the base layer features and the enhancement layer features; and
reconstructing the picture based on the combined features.
2 . The apparatus according to claim 1 , wherein the processing circuitry is further configured to execute the instructions to cause the apparatus to perform computer-vision processing based on the base layer features, and the computer-vision processing includes processing of the base layer features by one or more network layers of a first trained subnetwork.
3 . The apparatus according to claim 2 , wherein the reconstructing of the picture includes processing the combined features by one or more network layers of a second trained subnetwork different from the first trained subnetwork.
4 . The apparatus according to claim 1 , wherein the reconstructing of the picture includes:
reconstructing a base layer picture based on the base layer features; and
adding the enhancement layer features to the base layer picture.
5 . The apparatus according to claim 4 , wherein the enhancement layer features are based on differences between an encoder-side input picture and the base layer picture.
6 . The apparatus according to claim 1 , wherein the processing circuitry is configured to execute the instructions to cause the apparatus to perform the extracting the enhancement layer features from the enhancement layer bitstream and reconstructing the picture based on the base layer features and the enhancement layer features, and the reconstructed picture is a frame of a video, and the base layer features and the enhancement layer features are for a plurality of frames of the video.
7 . The apparatus according to claim 1 , wherein the processing circuitry is further configured to execute the instructions to further cause the apparatus to de-multiplex the base layer features and the enhancement layer features from a bitstream per frame.
8 . The apparatus according to claim 1 , wherein the processing circuitry is further configured to execute the instructions to further cause the apparatus to decrypt a portion of a bitstream including the enhancement layer features.
9 . An apparatus for processing a bitstream, the apparatus comprising:
a memory comprising instructions and a processing circuitry configured to execute the instructions to cause the apparatus to:
obtain a base layer bitstream including base layer features of a latent space and an enhancement layer bitstream including enhancement layer features;
extract from the base layer bitstream the base layer features; and
perform:
computer-vision processing based on the base layer features, including processing of the base layer features by one or more network layers of a first trained subnetwork.
10 . The apparatus according to claim 9 , wherein the processing circuitry is further configured to execute the instructions to cause the apparatus to perform extracting the enhancement layer features from the enhancement layer bitstream and reconstructing a picture based on the base layer features and the enhancement layer features, and the reconstructing of the picture includes:
combining the base layer features and the enhancement layer features; and
reconstructing the picture based on the combined features.
11 . The apparatus according to claim 10 , wherein the processing circuitry is further configured to execute the instructions to cause the apparatus to perform extracting the enhancement layer features from the enhancement layer bitstream and reconstructing the picture based on the base layer features and the enhancement layer features, and the reconstructing of the picture includes processing the combined features by one or more network layers of a second trained subnetwork different from the first trained subnetwork.
12 . The apparatus according to claim 9 , wherein the processing circuitry is further configured to execute the instructions to cause the apparatus to perform extracting the enhancement layer features from the enhancement layer bitstream and reconstructing a picture based on the base layer features and the enhancement layer features, and the reconstructing of the picture includes:
reconstructing a base layer picture based on the base layer features; and
adding the enhancement layer features to the base layer picture.
13 . The apparatus according to claim 12 , wherein the enhancement layer features are based on differences between an encoder-side input picture and the base layer picture.
14 . The apparatus according to claim 9 , wherein the processing circuitry is configured to execute the instructions to further cause the apparatus to perform extracting the enhancement layer features from the enhancement layer bitstream and reconstructing a picture based on the base layer features and the enhancement layer features, and the reconstructed picture is a frame of a video, and the base layer features and the enhancement layer features are for a plurality of frames of the video.
15 . The apparatus according to claim 9 , wherein the processing circuitry is further configured to execute the instructions to further cause the apparatus to de-multiplex the base layer features and the enhancement layer features from a bitstream per frame.
16 . The apparatus according to claim 9 , wherein the processing circuitry is further configured to execute the instructions to further cause the apparatus to decrypt a portion of a bitstream including the enhancement layer features.
17 . An apparatus for processing a bitstream, the apparatus comprising:
a memory comprising instructions and a processing circuitry configured to execute the instructions to cause the apparatus to:
obtain a base layer bitstream including base layer features of a latent space and an enhancement layer bitstream including enhancement layer features;
extract from the base layer bitstream the base layer features; and
perform:
extracting the enhancement layer features from the enhancement layer bitstream and reconstructing a picture based on the base layer features and the enhancement layer features, and the reconstructing of the picture comprises:
reconstructing a base layer picture based on the base layer features; and
adding the enhancement layer features to the base layer picture.
18 . The apparatus according to claim 17 , wherein the enhancement layer features are based on differences between an encoder-side input picture and the base layer picture.
19 . The apparatus according to claim 17 , wherein the processing circuitry is configured to execute the instructions to cause the apparatus to perform the extracting the enhancement layer features from the enhancement layer bitstream and reconstructing the picture based on the base layer features and the enhancement layer features, and the reconstructed picture is a frame of a video, and the base layer features and the enhancement layer features are for a plurality of frames of the video.
20 . The apparatus according to claim 17 , wherein the processing circuitry is further configured to execute the instructions to further cause the apparatus to de-multiplex the base layer features and the enhancement layer features from a bitstream per frame.
21 . The apparatus according to claim 17 , wherein the processing circuitry is further configured to execute the instructions to further cause the apparatus to decrypt a portion of a bitstream including the enhancement layer features.