Rate-distortion optimizers and optimization techniques including joint optimization of multiple color components
View Patent ↗Examples of encoders and video encoding are described that include optimizers and techniques for optimizing syntax elements such as transform coefficients. In some examples, multiple color components of a video signal may be jointly optimized by employing a cost calculation using a combination of distortion and/or rate metrics for multiple color components. In some examples, a color transformation may occur and the optimization may take place in a different color domain than encoding. In some examples, distortion metrics used in the cost calculations performed by optimizers are based on structural similarity index.
1. A video encoder comprising:
a transform circuit configured to transform a residual representation of a video signal in a spatial domain to a plurality of first transform coefficients in a frequency domain and a first color domain, wherein the video signal will be encoded in the first color domain; and
an optimizer circuit configured to (i) generate a plurality of additional transform coefficients in the frequency domain and the first color domain by interpolating the first transform coefficients, (ii) generate a plurality of second transform coefficients by converting the first transform coefficients and the additional transform coefficients from the first color domain to a second color domain, wherein (a) the second color domain is different than the first color domain and (b) the video signal will be displayed in the second color domain, (iii) generate a plurality of candidate transform coefficients from each one of the second transform coefficients and (iv) generate a plurality of optimized transform coefficients from the first transform coefficients and the additional transform coefficients using a cost calculation based on rate and distortion, wherein the cost calculation includes a combination of distortion metrics calculated from the second transform coefficients and the candidate transform coefficients.
2. The video encoder of claim 1 , further comprising an entropy encoder circuit configured to encode the optimized transform coefficients.
3. The video encoder of claim 1 , wherein the first color domain is YUV and the second color domain is RGB.
4. The video encoder of claim 1 , wherein the optimizer circuit comprises a color transform circuit configured to transform the first transform coefficients and the additional transform coefficients from the first color domain to the second color domain.
5. The video encoder of claim 4 , wherein the optimizer circuit further comprises an interpolator circuit configured to interpolate at least portions of the video signal.
6. The video encoder of claim 1 , wherein at least one of the distortion metrics is calculated based on a structural similarity index.
7. The video encoder of claim 6 , wherein at least one of the distortion metrics comprises a brightness metric based on a DC coefficient of the first transform coefficients.
8. The video encoder of claim 1 , wherein the optimizer circuit is further configured to perform the cost calculation including the combination of distortion metrics selectively based on object segmentation of the video signal.
9. The video encoder of claim 1 , wherein the optimizer circuit is further configured to perform the cost calculation including the combination of distortion metrics selectively based on resource availability in the video encoder.
10. The video encoder of claim 1 , wherein the combination of distortion metrics comprises a sum of distortions due to a plurality of chrominance components of the video signal.
11. A video encoder comprising:
a transform circuit configured to transform a residual representation of a video signal in a spatial domain to a plurality of first transform coefficients in a frequency domain and a first color domain, wherein the video signal will be encoded in the first color domain; and
an optimizer circuit configured to (i) generate a plurality of additional transform coefficients in the frequency domain and the first color domain by interpolating the first transform coefficients, (ii) generate a plurality of second transform coefficients by converting the first transform coefficients and the additional transform coefficients from the first color domain to a second color domain, wherein (a) the second color domain is different than the first color domain and (b) the video signal will be displayed in the second color domain, (iii) generate a plurality of candidate transform coefficients from each one of the second transform coefficients and (iv) generate a plurality of optimized transform coefficients from the first transform coefficients and the additional transform coefficients using a cost calculation based on a structural similarity index, wherein the cost calculation includes a combination of distortion metrics calculated from the second transform coefficients and the candidate transform coefficients.
12. The video encoder of claim 11 , wherein the cost calculation based on the structural similarity index comprises calculating a brightness distortion metric.
13. The video encoder of claim 12 , wherein the brightness distortion metric is based on a DC coefficient of the first transform coefficients.
14. The video encoder of claim 13 , wherein the cost calculation based on the structural similarity index comprises a calculation of a texture distortion metric based on at least one AC coefficient of the first transform coefficients.
15. The video encoder of claim 11 , wherein the optimizer circuit is further configured to (i) generate an optimized DC coefficient using the cost calculation based on the structural similarity index and (ii) generate a plurality of optimized AC coefficients using another cost calculation based on a sum of absolute differences or sum of square error.
16. A method comprising:
transforming a residual representation of a video signal in a spatial domain to a plurality of first transform coefficients in a frequency domain and a first color domain using a circuit, wherein the video signal will be encoded in the first color domain;
generating a plurality of additional transform coefficients in the frequency domain and the first color domain by interpolating the first transform coefficients;
generating a plurality of second transform coefficients by converting the first transform coefficients and the additional transform coefficients from the first color domain to a second color domain, wherein (a) the second color domain is different than the first color domain and (b) the video signal will be displayed in the second color domain;
generating a plurality of candidate transform coefficients from each one of the second transform coefficients;
calculating a rate-distortion cost using a combination of distortions calculated from the second transform coefficients and the candidate transform coefficients; and
generating a plurality of optimized transform coefficients from the first transform coefficients and the additional transform coefficients based on the rate-distortion costs, wherein the calculation of the rate-distortion cost includes a combination of distortion metrics calculated from the second transform coefficients and the candidate transform coefficients.
17. The method of claim 16 , further comprising:
calculating the rate-distortion costs using at least one distortion metric based on a structural similarity index.
18. The method of claim 16 , wherein the calculation of the rate-distortion costs comprises calculating costs for a plurality of luminance components of the video signal.
19. The method of claim 16 , wherein (i) the first color domain is used by an encoder configured to encode the optimized transform coefficients and (ii) the second color domain is used by a display configured to display a received video signal based on the encoded optimized transform coefficients.
20. The method of claim 16 , further comprising:
calculating the rate-distortion costs in the second color domain.
21. The method of claim 16 , further comprising:
selectively calculating the rate-distortion cost using the combination of distortions associated with certain objects identified in the video signal by object segmentation.