System and method for determining coding modes, DCT types and quantizers for video coding

1. A method for encoding a video including a sequence of frames, comprising:

measuring a variance of pixel intensities in a current frame;

assigning, according to rate and buffer fullness constraints, a number of bits to encode the current frame;

determining a multiplier value directly as a function of only the variance and the number of bits assigned to the current frame;

estimating motion vectors between a reference frame and the current frame;

determining a sum of absolute difference (SAD) based on a motion compensated residual between the reference frame and the current frame;

selecting an encoding mode for each macro block in the current frame based on the sum of absolute difference, the motion vectors and the multiplier value; and

encoding the motion compensated residual based on the encoding mode, multiplier value and the number of allocated bits.

2. The method of claim 1 , in which the encoding further comprises:

determining a quantization scale as a function of only the multiplier value and the number of bits assigned to the current frame;

extracting rate and distortion information associated with encoding each macro block in frame DCT mode and field DCT mode;

selecting a DCT type for each macro block in the current frame based on the multiplier value and the rate and distortion information;

transforming each macro block according to the selected DCT type;

quantizing each transformed macro block according to the selected quantizer; and

variable-length coding each quantized macro block as a bitstream.

3. The method of claim 1 , in which the multiplier value is λ=2ln2×σ 2 2 −2R , where R is the rate, and σ 2 is the variance.

4. The method of claim 1 , in which the selecting of the encoding mode further comprises:

minimizing a cost function cost=D+λR, where D is the distortion, R is the rate, λ is the multiplier;

modeling the distortion D by D(Q, SAD)=a×Q×SAD, where a is a constant coefficient; and

modeling the rate by R(Q,SAD)=MV+b×SAD/Q, where MV is an encoding rate for the motion vectors, and b is a constant coefficient.

5. The method of claim 2 , in which the selecting of the DCT type is based on the multiplier.

6. The method of claim 2 , in which the quantization scale is selected with a sliding window.