Granted Patent B2

US 12,256,092 · App. 18/192,343 · Granted Mar 18, 2025

Inventors: Amir Said (San Diego, CA); Hilmi Enes Egilmez (San Diego, CA); Marta Karczewicz (San Diego, CA); Vadim Seregin (San Diego, CA)

Assignee: QUALCOMM INCORPORATED

H04N19/45G06F17/16H04N19/117H04N19/12H04N19/147H04N19/176H04N19/18H04N19/463H04N19/61H04N19/625H04N19/96

Abstract

Claims (72)

1. A method of encoding video data comprising:

performing a first prediction process for a first block of video data to produce a first residual;

determining that a first transform process of a plurality of transform processes is to be applied to the first residual, wherein the first inverse transform process comprises one of a DCT-2 matrix, a DCT-3 matrix, a DST-2 matrix, or a DST-3 matrix;

applying the first transform process to the first residual to generate first transform coefficients for the first block of video data;

encoding the first transform coefficients;

performing a second prediction process for a second block of video data to produce a second residual;

determining that a second transform process is to be applied to the second residual, wherein the second transform process comprises the first transform process and at least one of a first pre-adjustment operation or a first post-adjustment operation to apply to the second residual in addition to the first transform process;

applying the first transform process and at least one of the first pre-adjustment operation or the first post-adjustment operation to the second residual to generate second transform coefficients for the second block of video data, wherein the first pre-adjustment operation, if applied, is applied prior to applying the first transform process, and wherein the first post- adjustment operation, if applied, is applied after applying the first transform process, and wherein the application of the first inverse transform process and the at least one of the pre- adjustment operation or the post-adjustment operation approximates an inverse transform process of a type different from the first inverse transform process; and

encoding the second transform coefficients.

2. The method of claim 1 , further comprising:

performing a third prediction process on the third block of video data to produce a third residual;

determining a subset of transform processes from the plurality of transform processes, wherein the subset of transform processes includes fewer transform processes than the plurality of transform processes, and wherein the subset of transform processes includes the first transform process;

determining a set of adjustment operations, wherein each adjustment operation comprises at least one of a pre-adjustment operation or a post-adjustment operation to be applied to the third residual, and wherein each adjustment operation, when applied in conjunction with a transform process of the subset of transform processes, results in transform coefficients that are approximately equal to transform coefficients resulting from the application of a transform process in the plurality of transform processes that is not included in the subset of transform processes, and wherein each adjustment operation of the set of adjustment operations is associated with a transform process of the subset of transform processes;

determining, for each transform process of the subset of transform processes, rate- distortion characteristics for the respective transform process;

determining, for each adjustment operation of the set of adjustment operations, rate- distortion characteristics for the respective adjustment operation and the associated transform process for the respective adjustment operation;

selecting, based on the determined rate-distortion characteristics, either the transform process from the subset of transform processes or the adjustment operation from the set of adjustment operations and the transform process from the subset of transform processes associated with the selected adjustment operation as a complete transform process to apply to the third residual;

applying the complete transform process to the third residual to generate third transform coefficients for the third block of video data; and

encoding the third transform coefficients.

3. The method of claim 2 , wherein the subset of transform processes comprises one or more of a DCT-2 matrix, a DCT-3 matrix, a DST-2 matrix, or a DST-3 matrix.

4. The method of claim 2 , further comprising:

encoding, in a bitstream, one or more indexes indicating the selected transform process from the subset of transform processes; and

if an adjustment operation is selected, encoding, in the bitstream, one or more indexes indicating the selected adjustment operation from the set of adjustment operations.

5. The method of claim 1 , wherein encoding the first transform coefficients and encoding the second transform coefficients comprises entropy encoding the first transform coefficients and entropy encoding the second transform coefficients.

6. The method of claim 5 , wherein each transform process of the plurality of transform processes comprises a discrete trigonometrical transform matrix.

7. The method of claim 1 , wherein the first transform process comprises a discrete trigonometrical transform matrix.

8. The method of claim 1 , wherein the first transform process and the at least one of the first pre-adjustment operation or the first post-adjustment operation each comprise a respective sparse matrix.

9. The method of claim 8 , wherein the sparse matrix of the first transform process comprises a band diagonal matrix.

10. The method of claim 8 , wherein the sparse matrix of the first transform process comprises a block diagonal matrix.

11. The method of claim 1 , wherein the at least one of the first pre-adjustment operation or the first post-adjustment operation comprises a set of one or more Givens rotations.

12. The method of claim 1 , wherein the first block of video data comprises a first coding unit in a first row of a coding tree unit, wherein the second block of video data comprises a second coding unit in a second row of the coding tree unit, and wherein the first row is different than the second row.

13. The method of claim 1 , wherein the first block of video data comprises a first coding unit in a first column of a coding tree unit, wherein the second block of video data comprises a second coding unit in a second column of the coding tree unit, and wherein the first column is different than the second column.

14. The method of claim 1 , wherein the first block of video data comprises a first coding unit in a first row of a coding tree unit, wherein the second block of video data comprises a second coding unit in a second row of the coding tree unit, wherein the first row is different than the second row, and wherein the method further comprises:

performing a third prediction process for a third coding unit of the coding tree to produce a third residual, wherein the third coding unit is in a third row of the coding tree unit, wherein the second row and the third row are contiguous rows;

determining that the second transform process is to be applied to the third residual;

applying the first transform process and at least one of the first pre-adjustment operation or the first post-adjustment operation to the third residual to generate third transform coefficients for the third coding unit; and

encoding the third transform coefficients.

15. The method of claim 1 , wherein the first block of video data comprises a first coding tree unit, and wherein the second block of video data comprises a second coding tree unit different than the first coding tree unit.

16. A video encoding device comprising:

a memory configured to store video data; and

one or more processors implemented in circuitry and configured to:

perform a first prediction process for a first block of video data to produce a first residual;

determine that a first transform process of a plurality of transform processes is to be applied to the first residual, wherein the first inverse transform process comprises one of a DCT-2 matrix, a DCT-3 matrix, a DST-2 matrix, or a DST-3 matrix;;

apply the first transform process to the first residual to generate first transform coefficients for the first block of video data;

encode the first transform coefficients;

determine that a second transform process is to be applied to the second residual, wherein the second transform process comprises the first transform process and at least one of a first pre-adjustment operation or a first post-adjustment operation to apply to the second residual in addition to the first transform process;

apply the first transform process and at least one of the first pre-adjustment operation or the first post-adjustment operation to the second residual to generate second transform coefficients for the second block of video data, wherein the first pre-adjustment operation, if applied, is applied prior to applying the first transform process, and wherein the first post-adjustment operation, if applied, is applied after applying the first transform process and wherein the application of the first inverse transform process and the at least one of the pre-adjustment operation or the post-adjustment operation approximates an inverse transform process of a type different from the first inverse transform process; and

encode the second transform coefficients.

17. The video encoding device of claim 16 , wherein the one or more processors are further configured to:

perform a third prediction process on the third block of video data to produce a third residual;

determine a subset of transform processes from the plurality of transform processes, wherein the subset of transform processes includes fewer transform processes than the plurality of transform processes, and wherein the subset of transform processes includes the first transform process;

determine a set of adjustment operations, wherein each adjustment operation comprises at least one of a pre-adjustment operation or a post-adjustment operation to be applied to the third residual, and wherein each adjustment operation, when applied in conjunction with a transform process of the subset of transform processes, results in transform coefficients that are approximately equal to transform coefficients resulting from the application of a transform process in the plurality of transform processes that is not included in the subset of transform processes, and wherein each adjustment operation of the set of adjustment operations is associated with a transform process of the subset of transform processes;

determine, for each transform process of the subset of transform processes, rate-distortion characteristics for the respective transform process;

determine, for each adjustment operation of the set of adjustment operations, rate- distortion characteristics for the respective adjustment operation and the associated transform process for the respective adjustment operation;

select, based on the determined rate-distortion characteristics, either the transform process from the subset of transform processes or the adjustment operation from the set of adjustment operations and the transform process from the subset of transform processes associated with the selected adjustment operation as a complete transform process to apply to the third residual;

apply the complete transform process to the third residual to generate third transform coefficients for the third block of video data; and encode the third transform coefficients.

18. The video encoding device of claim 17 , wherein the subset of transform processes comprises one or more of a DCT-2 matrix, a DCT-3 matrix, a DST-2 matrix, or a DST-3 matrix, wherein the one or more processors are further configured to:

encode, in a bitstream, one or more indexes indicating the selected transform process from the subset of transform processes; and

if an adjustment operation is selected, encode, in the bitstream, one or more indexes indicating the selected adjustment operation from the set of adjustment operations.

19. The video encoding device of claim 16 , wherein encoding the first transform coefficients and encoding the second transform coefficients comprises entropy encoding the first transform coefficients and entropy encoding the second transform coefficients.

20. The video encoding device of claim 16 , wherein the first transform process comprises a discrete trigonometrical transform matrix, and wherein each transform process of the plurality of transform processes comprises a discrete trigonometrical transform matrix.

21. The video encoding device of claim 16 , wherein the first transform process and the at least one of the first pre-adjustment operation or the first post-adjustment operation each comprise a respective sparse matrix, wherein the sparse matrix of the first transform process comprises one of a band diagonal matrix or a block diagonal matrix.

22. The video encoding device of claim 16 , wherein the at least one of the first pre-adjustment operation or the first post-adjustment operation comprise a set of one or more Givens rotations.

23. The video encoding device of claim 16 , wherein the first block of video data comprises a first coding unit in a first row of a coding tree unit, wherein the second block of video data comprises a second coding unit in a second row of the coding tree unit, and wherein the first row is different than the second row.

24. The video encoding device of claim 16 , wherein the first block of video data comprises a first coding unit in a first column of a coding tree unit, wherein the second block of video data comprises a second coding unit in a second column of the coding tree unit, and wherein the first column is different than the second column.

25. The video encoding device of claim 16 , wherein the first block of video data comprises a first coding unit in a first row of a coding tree unit, wherein the second block of video data comprises a second coding unit in a second row of the coding tree unit, wherein the first row is different than the second row, and wherein the one or more processors are further configured to:

predict a third residual for a third coding unit of the coding tree unit, wherein the third coding unit is in a third row of the coding tree unit, wherein the second row and the third row are contiguous rows;

determine that the second transform process is to be applied to the third residual;

apply the first transform process and at least one of the first pre-adjustment operation or the first post-adjustment operation to the third residual to generate third transform coefficients for the third coding unit; and

encode the third transform coefficients.

26. The video encoding device of claim 16 , wherein the first block of video data comprises a first coding tree unit, and wherein the second block of video data comprises a second coding tree unit different than the first coding tree unit.

27. The video encoding device of claim 16 , further comprising:

a camera configured to capture the video data.

Assignments (1)

Continuity (4)

References Cited (52)