Integrated image reshaping and video coding
Given a sequence of images in a first codeword representation, methods, processes, and systems are presented for integrating reshaping into a next generation video codec for encoding and decoding the images, wherein reshaping allows part of the images to be coded in a second codeword representation which allows more efficient compression than using the first codeword representation. A variety of architectures are discussed, including: an out-of-loop reshaping architecture, an in-loop-for intra pictures only reshaping architecture, an in-loop architecture for prediction residuals, and a hybrid in-loop reshaping architecture. Syntax methods for signaling reshaping parameters, and image-encoding methods optimized with respect to reshaping are also presented.
1. An apparatus for encoding images, the apparatus comprising:
an input for accessing an input image in a first codeword representation; and a processor, wherein the processor:
generates a forward reshaping function mapping pixels of the input image to a second codeword representation;
generates an inverse reshaping function based on the forward reshaping function, wherein the inverse reshaping function maps pixels from the second coding representation to the first coding representation;
generates coded pixel regions of the input image based on input pixel regions in the input image, the forward reshaping function and the inverse reshaping function;
generates reshaping metadata which characterize the forward reshaping function based on a piecewise-linear representation; and
generates an output bitstream based on the coded pixel regions of the input image and the reshaping metadata.
2. The apparatus of claim 1 , wherein to generate the coded pixel regions of the input image the processor applies in-loop reshaping.
3. The apparatus of claim 1 , wherein for an input pixel region in the input image, to generate a coded pixel region, the processor:
computes a predicted region based on pixel data in a reference frame buffer or previously coded spatial neighbors;
generates a reshaped residual region based on the input pixel region, the predicted region, and the forward reshaping function, wherein a reshaped residual sample in the reshaped residual region is derived at least in part from forward reshaping a respective predicted sample in the predicted region;
generates a quantized residual region based on the reshaped residual region;
generates a dequantized residual region based on the quantized residual region;
generates a reconstructed pixel region based on the dequantized residual region, the predicted region, the forward reshaping function, and the inverse reshaping function; and
generates a reference pixel region to be stored on the reference frame buffer based on the reconstructed pixel region.
4. The apparatus of claim 3 , wherein generating the quantized residual region comprises:
applying a forward coding transform to the reshaped residual region to generate transformed data; and
applying a forward coding quantizer to the transformed data to generate quantized data.
5. The apparatus of claim 4 , wherein generating the dequantized residual region comprises:
applying an inverse coding quantizer to the quantized data to generate inverse-quantized data; and
applying an inverse coding transform to the inverse-quantized data to generate the dequantized residual region.
6. The apparatus of claim 3 , wherein generating the reference pixel region to be stored on the reference frame buffer comprises applying a loop filter to the reconstructed pixel region.
7. The apparatus of claim 3 , wherein generating the reshaped residual region comprises computing:
Res_ r ( i )=Fwd(Orig_sample( i ))−Fwd(Pred_sample( i )),
where Fwd( ) denotes the forward reshaping function, Res_r(i) denotes a pixel of the reshaped residual region, Orig_sample(i) denotes a pixel of the input image region, and Pred_sample(i) denotes a pixel of the predicted region.
8. The apparatus of claim 7 , wherein generating the reconstructed pixel region comprises computing:
Recon_sample( i )=Inv(Res_ d ( i )+Fwd(Pred_sample( i ))),
where Inv( ) denotes the inverse reshaping function, Recon_sample (i) denotes a pixel of the reconstructed pixel region and Res_d(i) denotes a pixel of the dequantized residual region representing a close approximation of the Res_r(i) pixel.
9. The apparatus of claim 1 , wherein for each segment of a piecewise linear representation of the forward reshaping function, the reshaping metadata comprise a delta absolute value and a sign of the delta absolute value.
10. The apparatus of claim 9 , wherein for bin_ce_delta_abs[i] denoting the delta absolute value for segment i, its value represents the difference of codewords allocated in segment i compared to the codewords allocated in segment i−1 (bin_ce_delta_abs[i−1]).
11. An apparatus for decoding a coded bitstream to generate an output image in a first codeword representation, the apparatus comprising:
an input to receive a coded image partially coded in a second codeword representation; and a processor, wherein the processor:
receives reshaping metadata for the coded image;
generates based on the reshaping metadata the forward reshaping function mapping pixels from the first codeword representation to the second codeword representation;
generates based on the reshaping metadata an inverse reshaping function, wherein the inverse reshaping function maps pixels from the second codeword representation to the first codeword representation; and
decodes coded regions of the coded image based on the forward reshaping function and the inverse reshaping function to generate output pixel regions.
12. The apparatus of claim 11 , wherein the processor decodes the coded regions of the coded image based on in-loop reshaping.
13. The apparatus of claim 11 , wherein to decode a coded region, the processor:
generates a decoded reshaped residual region;
generates a predicted region based on pixels in a reference pixel buffer or previously decoded spatial neighbors;
generates a reconstructed pixel region based on the decoded reshaped residual region, the predicted region, the forward reshaping function, and the inverse reshaping function, wherein a reconstructed sample in the reconstructed pixel region is derived at least in part from forward reshaping a respective predicted sample in the predicted region;
generates an output pixel region for the output image based on the reconstructed pixel region; and
stores the output pixel region in the reference pixel buffer.
14. The apparatus of claim 12 , wherein generating the reconstructed pixel region comprises computing:
Reco_sample( i )=Inv(Res_ d ( i )+Fwd(Pred_sample( i ))),
where Reco_sample (i) denotes a pixel of the reconstructed pixel region, Res_d(i) denotes a pixel of the decoded reshaped residual region, Inv( ) denotes the inverse reshaping function, Fwd( ) denotes the forward reshaping function, and Pred_sample(i) denotes a pixel of the predicted region.
15. The apparatus of claim 11 , wherein the metadata characterize the forward reshaping function based on a piecewise linear representation, and for each segment of the piecewise linear representation of the forward reshaping function, the reshaping metadata comprise a delta absolute value and a sign of the delta absolute value.
16. The apparatus of claim 15 , wherein for bin_ce_delta_abs[i] denoting the delta absolute value for segment i, its value represents the difference of codewords allocated in segment i compared to the codewords allocated in segment i−1 (bin_ce_delta_abs[i−1]).