Method and encoder for constrained soft-decision quantization in data compression

1. A method of encoding a video, the video including a block of pixel data, the pixel data for the block having been transformed to a set of transform domain coefficients, the method comprising:

quantizing the set of transform domain coefficients using a quantization step size to obtain a set of hard quantized coefficients, wherein each hard quantized coefficient is an integer-value hard-quantization level to which the coefficient has been quantized;

for each of the transform domain coefficients in the set of transform domain coefficients, without changing the quantization step size or other quantization parameters, performing soft decision quantization to obtain a soft quantized coefficient using a rate-distortion calculation over a search range of quantization levels for that transform domain coefficient, wherein the search range of quantization levels for that transform domain coefficient is a range of integers around the integer-value hard-quantization level and wherein that range is constrained within a number of quantization levels of the hard quantized coefficient, and wherein the number of quantization levels is based on a threshold value less accumulated differences between previous soft quantized coefficients for the set of transform domain coefficients and corresponding hard quantized coefficients; and

entropy encoding the soft quantized coefficients to generate encoded video data.

2. The method claimed in claim 1 , wherein the accumulated differences comprises an L 1 norm between a vector of the previous soft quantized coefficients and a vector of the corresponding hard quantized coefficients.

3. The method claimed in claim 1 , further including building a trellis of costs, wherein the trellis includes a stage for each of the transform domain coefficients, and each stage includes a state for each unique set of coding scheme parameters, and wherein building the trellis includes making connections between states when valid in accordance with the coding scheme, and wherein performing soft decision quantization includes calculating rate-distortion costs for each state in the trellis and selecting a desired quantization level for each state on the basis of realizing a minimum cumulative rate-distortion cost for that state.

4. The method claimed in claim 3 , wherein performing soft-decision quantization includes limiting the calculation of rate-distortion costs for each state on the basis of the search range.

5. The method claimed in claim 4 , further comprising tracing back through the trellis from an end state to the beginning on the basis of realizing a minimum cumulative rate-distortion cost to obtain the soft quantized coefficients.

6. The method claimed in claim 1 , wherein the rate-distortion calculation for a given quantization level includes determining distortion associated with the given quantization level and determining the coding rate impact for the given quantization levels based on a transition from an earlier quantization levels of a previous coefficients in the set of transform domain coefficients.

7. The method claimed in claim 6 , wherein the coding rate impact is based on the coding syntax of the coding scheme to be used for entropy encoding.

8. The method claimed in claim 1 , wherein the number of quantization levels is based on a position of that transform domain coefficient in the set of transform domain coefficients.

9. The method claimed in claim 1 , wherein the number of quantization levels for the search range for that transform domain coefficient is based on the magnitude of the corresponding hard quantized coefficient for that transform domain coefficient.

10. An encoder for encoding a video, the encoder comprising:

a processor;

a memory; and

an encoding application stored in memory and containing instructions for configuring the processor to encode the video by

quantizing the set of transform domain coefficients using a quantization step size to obtain a set of hard quantized coefficients, wherein each hard quantized coefficient is an integer-value hard-quantization level to which the coefficient has been quantized;

for each of the transform domain coefficients in the set of transform domain coefficients, without changing the quantization step size or other quantization parameters, performing soft decision quantization to obtain a soft quantized coefficient using a rate-distortion calculation over a search range of quantization levels for that transform domain coefficient, wherein the search range of quantization levels for that transform domain coefficient a range of integers around the integer-value hard-quantization level and wherein that range is constrained within a number of quantization levels of the hard quantized coefficient, and

entropy encoding the soft quantized coefficients to generate encoded video data,

and wherein the number of quantization levels is based on a threshold value less accumulated differences between previous soft quantized coefficients for the set of transform domain coefficients and corresponding hard quantized coefficients.

11. The encoder claimed in claim 10 , wherein the accumulated differences comprises an L 1 norm between a vector of the previous soft quantized coefficients and a vector of the corresponding hard quantized coefficients.

12. The encoder claimed in claim 11 , wherein the encoding application further configures the processor to build a trellis of costs, wherein the trellis includes a stage for each of the transform domain coefficients, and each stage includes a state for each unique set of coding scheme parameters, and wherein the processor is further configured to make connections between states when valid in accordance with the coding scheme, and wherein the processor is further configured to perform soft decision quantization by calculating rate-distortion costs for each state in the trellis and selecting a desired quantization level for each state on the basis of realizing a minimum cumulative rate-distortion cost for that state.

13. The encoder claimed in claim 12 , wherein the processor is further configured to perform soft-decision quantization by limiting the calculation of rate-distortion costs for each state on the basis of the search range.

14. The encoder claimed in claim 13 , and wherein the processor is further configured to trace back through the trellis from an end state to the beginning on the basis of realizing a minimum cumulative rate-distortion cost to obtain the soft quantized coefficients.

15. The encoder claimed in claim 10 , wherein the processor is further configured to calculate the rate-distortion for a given quantization level by determining distortion associated with the given quantization level and determining the coding rate impact for the given quantization levels based on a transition from an earlier quantization levels of a previous coefficients in the set of transform domain coefficients.

16. The encoder claimed in claim 15 , wherein the coding rate impact is based on the coding syntax of the coding scheme to be used for entropy encoding.

17. The encoder claimed in claim 10 , wherein the number of quantization levels is based on a position of that transform domain coefficient in the set of transform domain coefficients.

18. The encoder claimed in claim 10 , wherein the number of quantization levels for the search range for that transform domain coefficient is based on the magnitude of the corresponding hard quantized coefficient for that transform domain coefficient.

19. A non-transitory computer-readable medium having stored thereon computer-executable instructions which, when executed by a processor, configure the processor to execute the method claimed in claim 1 .