IP Library Granted Patent US 9,747,251
Granted Patent B2
US 9,747,251 · App. 13/314,013 · Granted Aug 29, 2017

Video encoding and decoding using parallel processors

Inventors: Jesse J. Rosenzweig (Portland, OR); Brian Gregory Lewis (Portland, OR)
Assignee: AMAZON TECHNOLOGIES, INC.
G06F15/8007H04N19/174H04N19/176H04N19/436H04N19/44H04N19/91
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 9,747,251
App. No.
13/314,013
Granted
Aug 29, 2017
Kind
B2
Abstract

A method is disclosed for the decoding and encoding of a block-based video bit-stream such as MPEG2, H.264-AVC, VC1, or VP6 using a system containing one or more high speed sequential processors, a homogenous array of software configurable general purpose parallel processors, and a high speed memory system to transfer data between processors or processor sets. This disclosure includes a method for load balancing between the two sets of processors.

Claims (72)

1. A system for decompressing a compressed video data stream, the system comprising:

a sequential processor array comprising at least one sequential processor and arranged to receive a video data input stream, the sequential processor array configured to recover macroblock coefficient data and macroblock metadata from a plurality of portions of the video data input stream;

a parallel processor array (PPA) comprising a plurality of parallel multi-processors, the PPA configured to process the recovered macroblock coefficient data and macroblock to recover pixels;

a data bus interconnecting the sequential processor array and the PPA;

a first RAM memory coupled to the sequential processor array to store the macroblock coefficient data and the macroblock metadata, the macroblock metadata including one or more properties for determining to process a respective macroblock using at least one of a Parallel Variable Length Decode (PVLD) process or a Parallel Block Decode (PBD) process;

a second RAM memory coupled to the PPA to store pixel data output from the PPA, the data bus configured to transfer the macroblock coefficient data and the macroblock metadata from the first RAM memory to the second RAM memory; and

an index array including a plurality of pointers corresponding to the plurality of portions of the video data input stream, wherein the sequential processor array processes the video data input stream using the index array.

2. The system of claim 1 , wherein the second RAM is arranged for storing PPA program instructions and the macroblock coefficient data, the macroblock metadata, reference frame data, and output video frame data.

3. The system of claim 1 , wherein the multi-processors comprise SIMD processors.

4. The system of claim 1 , wherein:

the first RAM and the second RAM are realized in a single shared RAM; and the PPA has access to the shared RAM to read the macroblock coefficient data and the macroblock metadata.

5. The system of claim 1 , wherein the PVLD process comprises:

decompressing a first video frame of the video data input stream;

assigning a first processing array to the plurality of portions of the video data input stream using the index array;

producing a first independent variable length coefficient data set responsive to the first video frame;

producing a first fixed-size metadata data set responsive to the first video frame;

storing the first independent variable length coefficient data set and the first fixed-size metadata data set in the first RAM memory;

transferring the first independent variable length coefficient data set and the first fixed-size metadata data set from the first RAM memory to a second RAM memory over a data bus interconnecting the first processing array and a second processing array that is different from the first processing array; and

decompressing a second video frame of the video data input stream.

6. The system of claim 1 , wherein:

the sequential processor array comprises a single semiconductor integrated circuit; and

the parallel processor array comprises a second semiconductor integrated circuit.

7. The system of claim 5 , wherein the PBD process is executed in parallel with the PVLD process executing on the video data input stream, the PBD process comprising:

accessing the second RAM memory coupled to the second processing array;

decompressing, using the second processing array, the respective macroblock of the video data input stream by decoding the first independent variable length coefficient data set using the first fixed-size metadata data set and using at least one previously-stored reference frame: and

generating a desired output video frame based on decompressing the respective macroblock, wherein the PBD process is executed on the respective macroblock prior to a complete execution of the PVLD process on the video data input stream.

8. A method for decompressing a variable length inter-block dependent compressed video input data stream wherein the video input data stream comprises macroblocks, the method comprising:

determining one or more properties of metadata for a respective macroblock of the video input data stream; and

executing a Parallel Variable Length Decode (PVLD) process on the video input data stream using a first processing array coupled to a first RAM memory based at least in part on the one or more properties of the metadata of the respective macroblock, the PVLD process comprising:

decompressing a first video frame of the video input data stream;

assigning the first processing array to a plurality of portions of the first video frame using an index array, the index array including a plurality of pointers corresponding to the plurality of portions;

producing a first independent variable length coefficient data set responsive to the first video frame;

producing a first fixed-size metadata data set responsive to the first video frame;

storing the first independent variable length coefficient data set and the first fixed-size metadata data set in the first RAM memory;

transferring the first independent variable length coefficient data set and the first fixed-size metadata data set from the first RAM memory to a second RAM memory over a data bus interconnecting the first processing array and a second processing array; and

decompressing a second video frame of the video input data stream.

9. The method of claim 8 , further comprising:

using a central processing unit (CPU) component of a computing device, recovering macroblock coefficient data and the macroblock metadata from the video input data stream;

providing the recovered macroblock coefficient data and the recovered macroblock metadata from the CPU component to a Graphics Processing Unit (GPU) component of the computing device over an internal bus of the computing device; and

using the GPU component, processing the recovered macroblock coefficient data and the recovered macroblock metadata from the CPU component to recover pixels.

10. The method of claim 9 , wherein providing the recovered macroblock coefficient data and the recovered macroblock metadata from the CPU component to the component of the computing device further comprises transmitting the recovered macroblock coefficient data and the recovered macroblock metadata over a bus that interconnects two separate semiconductor integrated circuits.

11. The method of claim 8 , further comprising:

executing a Parallel Block Decode (PBD) process using the second processing array based at least in part on the one or more properties of the metadata for the respective macroblock, the second processing array different from the first processing array and coupled to the second RAM memory, the PBD process executing in parallel with the PVLD process executing on the video input data stream, the PBD process comprising:

decompressing the respective macroblock of the video input data stream by decoding the stored first independent variable length coefficient data set using the stored first fixed-size metadata data set and using at least one previously-stored reference frame, so as to generate a desired output video frame; and

commencing execution of the PVLD process on the second frame of video input data stream following completion of said storing of the first video frame to cause the PVLD process to decompress at least one frame ahead of the PBD process.

12. The method of claim 11 , wherein the PBD process decodes all of the macroblocks of a current frame concurrently.

13. The method of claim 11 , wherein said the first independent variable length coefficient data set and the first fixed-size metadata data set is stored in a shared memory space accessible to an array of multi-processors.

14. The method of claim 11 , further comprising:

encoding frequency coefficient data using a selected run length coding;

wherein the frequency coefficient data represents residual data generated from predicting a next macroblock of the video input data stream.

15. The method of claim 11 , wherein the first fixed-sized metadata data set contains properties of the respective macroblock that instruct the second processing array for decoding the respective macroblock.

16. The method of claim 15 , wherein the properties of the respective macroblock stored in the first fixed-size metadata data set that instruct multi-processors in a parallel processor array (PPA) for decoding the respective macroblock include at least coded block pattern, prediction modes, quantization parameter, and motion vectors.

17. A method for decompressing a variable length inter-block dependent encoded video input data stream wherein the variable length inter-block dependent encoded video input data stream comprises macroblocks, the method comprising:

determining one or more properties of metadata for a respective macroblock; and

executing a Parallel Variable Length Decode (PVLD) process on the variable length inter-block dependent encoded video input data stream using a first processing array coupled to a first RAM memory based at least in part on the one or more properties of the metadata of the respective macroblock, the PVLD process comprising:

decompressing a first video frame of the variable length inter-block dependent encoded video input data stream;

assigning the first processing array to a plurality of portions of the first frame using an index array, the index array including a plurality of pointers corresponding to the plurality of portions;

producing a first macroblock coefficient data set responsive to the first video frame;

producing a first macroblock metadata data set responsive to the first video frame;

storing the first macroblock coefficient data set and the first macroblock metadata data set in the first RAM memory;

transferring the first macroblock coefficient data set and the first macroblock metadata data set from the first RAM memory to a second RAM memory over a data bus interconnecting the first processing array and a second processing array; and

decompressing a second video frame of the variable length inter-block dependent encoded video input data stream.

18. The method of claim 17 , further comprising:

executing the PBD process using the second processing array based at least in part on the one or more properties of the metadata of the respective macroblock, the second processing array different than the first processing array and coupled to the second RAM memory, the PBD process executing in parallel with the PVLD process executing on the variable length inter-block dependent encoded video input data stream, the PBD process comprising:

decompressing the macroblocks of a previous frame of the variable length inter-block dependent encoded video input data stream; and

executing the PVLD process to decompress at least one frame ahead of the PBD process.

19. The method of claim 17 , further comprising:

using a central processing unit (CPU) component of a computing device, recovering the first macroblock coefficient data set and the first macroblock metadata set from the variable length inter-block dependent encoded video input data stream;

providing the recovered macroblock coefficient data set and the recovered macroblock metadata set from the CPU component to a Graphics Processing Unit (GPU);

component of the computing device over an internal bus of the computing device; and

using the GPU component, processing the recovered macroblock coefficient data set and the recovered macroblock metadata set from the CPU component to recover pixels.

20. The method of claim 19 , wherein providing the recovered macroblock coefficient data and the recovered macroblock metadata from the CPU component to the component of the computing device further comprises transmitting the recovered macroblock coefficient data set and the recovered macroblock metadata set over a bus that interconnects two separate semiconductor integrated circuits.

Assignments (2)
BILL OF SALE Recorded Jan 12, 2016
From: ELEMENTAL TECHNOLOGIES LLC
To: AMAZON TECHNOLOGIES, INC.
Reel/Frame 037486/0985 →
CERTIFICATE OF CONVERSION Recorded Jan 12, 2016
From: ELEMENTAL TECHNOLOGIES, INC.
To: ELEMENTAL TECHNOLOGIES LLC
Reel/Frame 037486/0995 →
Continuity (3)
Continuation 12058636 · Mar 28, 2008
Provisional Application 61002972 · Nov 13, 2007
Related Publication 20120093234A1 · Apr 19, 2012