Video decoding implementations for a graphics processing unit
Video decoding innovations for multithreading implementations and graphics processor unit (“GPU”) implementations are described. For example, for multithreaded decoding, a decoder uses innovations in the areas of layered data structures, picture extent discovery, a picture command queue, and/or task scheduling for multithreading. Or, for a GPU implementation, a decoder uses innovations in the areas of inverse transforms, inverse quantization, fractional interpolation, intra prediction using waves, loop filtering using waves, memory usage and/or performance-adaptive loop filtering. Innovations are also described in the areas of error handling and recovery, determination of neighbor availability for operations such as context modeling and intra prediction, CABAC decoding, computation of collocated information for direct mode macroblocks in B slices, reduction of memory consumption, implementation of trick play modes, and picture dropping for quality adjustment.
1. A computer-readable medium storing computer-executable instructions for causing a computer system, when programmed thereby, to perform video processing, wherein the computer-readable medium is a magnetic disk or non-volatile memory, the video processing comprising:
receiving a coded video bit stream including plural pictures in serial coded order; and
decoding the plural pictures with one or more central processing units (“CPUs”) and a graphics processing unit (“GPU”), wherein the one or more CPUs perform plural decoding operations, and wherein the GPU performs plural decoding operations asynchronously from the plural decoding operations performed by the one or more CPUs, including:
with at least one of the one or more CPUs, filling a GPU command buffer with tasks and markers for processing by the GPU, including:
generating the tasks, for the plural decoding operations performed by the GPU, and inserting the tasks in the GPU command buffer for execution by the GPU; and
generating the markers, which are different than the tasks for the plural decoding operations performed by the GPU, and inserting the markers in the GPU command buffer, each of the markers indicating completion of decoding of a corresponding picture of the plural pictures; and
upon triggering of one of the markers in the GPU command buffer, marking the corresponding picture for the marker as available for reference in a decoded picture buffer (“DPB”);
wherein the one or more CPUs and the GPU use the GPU command buffer to track completion of the decoding of the plural pictures, respectively, and manage the DPB.
2. The computer-readable medium of claim 1 wherein the plural decoding operations performed by the one or more CPUs include picture extent discovery, and wherein the plural decoding operations performed by the GPU include intra prediction, motion compensation, and loop filtering.
3. The computer-readable medium of claim 1 wherein, for a given picture of the plural pictures, the GPU performs at least some of its decoding operations in parallel for plural blocks of the given picture.
4. The computer-readable medium of claim 1 wherein the decoding the plural pictures further comprises, with the GPU, processing the tasks and the markers in serial order.
5. The computer-readable medium of claim 1 wherein the decoding the plural pictures further comprises:
with the GPU, triggering one of the markers upon reaching the marker in the GPU command buffer.
6. The computer-readable medium of claim 5 wherein, upon the triggering of the marker, the corresponding picture is copied into an output buffer.
7. The computer-readable medium of claim 1 wherein the decoding the plural pictures further comprises:
representing a reference picture as a texture in memory; and
using texture operations to access the reference picture during motion compensation.
8. The computer-readable medium of claim 1 wherein the decoding the plural pictures further comprises:
using the GPU command buffer to track plural memory partition patterns, the tracked plural memory partition patterns including at least some inconsistent memory partition patterns in flight during at least part of the decoding the plural pictures.
9. The computer-readable medium of claim 1 wherein the decoding the plural pictures further comprises:
using the GPU command buffer to track plural reference picture slot assignment patterns, the tracked plural reference picture slot assignment patterns including at least some inconsistent reference picture slot assignment patterns in flight during at least part of the decoding the plural pictures.
10. A method comprising:
receiving a coded video bit stream that includes encoded video for plural pictures in serial coded order;
scanning ahead in the coded video bit stream to determine plural resource usage patterns for the plural pictures;
tracking the plural resource usage patterns, the tracked plural resource usage patterns including at least some inconsistent resource usage patterns in flight during at least part of decoding, wherein the tracking the plural resource usage patterns includes:
with one or more central processing units (“CPUs”), before decoding of the plural pictures by a graphics processing unit (“GPU”) begins, for each of at least some of the plural pictures, prospectively assigning a slot of a reference picture array that will store the picture for use as a reference picture after decoding of the picture by the GPU; and
performing decoding operations on the plural pictures in serial coded order with the GPU, including storing the at least some of the plural pictures, respectively, at the prospectively assigned slots of the reference picture array.
11. The method of claim 10 wherein the performing decoding operations includes, for a given picture of the plural pictures, performing one or more of the decoding operations on plural blocks of the given picture in parallel.
12. The method of claim 10 wherein each of the at least some inconsistent resource usage patterns is valid during part of the decoding and invalid during a different part of the decoding.
13. The method of claim 10 further comprising tracking commands for the decoding operations for the plural pictures in a GPU command queue, wherein the GPU command queue also represents the plural resource usage patterns.
14. The method of claim 10 wherein the inconsistent resource usage patterns in flight include two or more different reference picture slot assignment patterns for the reference picture array.
15. The method of claim 10 wherein the plural resource usage patterns further include plural memory partition patterns for the reference picture array, and wherein the inconsistent resource usage patterns in flight further include two or more different memory partition patterns for the reference picture array, the two or more different memory partition patterns corresponding to different formats for the reference pictures.
16. A computer system comprising:
a decoder configured to perform organizing plural blocks of a picture as plural waves that correspond to diagonal lines of blocks among the plural blocks of the picture, each of the plural waves including one or more of the plural blocks such that block-to-block dependencies are not permitted within a given wave of the plural waves but are permitted between the given wave and any preceding waves of the plural waves; and
a graphics processing unit (“GPU”) configured to perform decoding operations on the plural blocks on a wave-by-wave basis, including for at least one of the plural waves processing some of the one or more blocks within the wave in parallel.
17. The computer system of claim 16 wherein the plural blocks are plural intra blocks, and wherein the decoding operations are intra prediction operations.
18. The computer system of claim 16 wherein the decoding operations are loop filtering operations.
19. The computer system of claim 16 wherein the organizing includes:
grouping, as a first wave of the plural waves, a first set of one or more of the plural blocks having no dependencies on other blocks of the plural blocks;
grouping, as a second wave of the plural waves, a second set of one or more of the plural blocks having no dependencies other than dependencies on the first set; and
grouping, as a third wave of the plural waves, a third set of one or more of the plural blocks having no dependencies other than dependencies on the first and second sets.
20. The computer system of claim 16 wherein the organizing is based upon static assumptions of dependencies for the plural blocks.
21. The computer system of claim 16 wherein the organizing is based upon actual dependencies for the plural blocks.