Memory word array organization and prediction combination for memory access
View Patent ↗Described systems and methods allow a reduction in the memory bandwidth required in video coding (decoding/encoding) applications. According to a first aspect, the data assigned to each memory word is chosen to correspond to a 2D subarray of a larger array such as a macroblock. An array memory word organization allows reducing both the average and worst-case bandwidth required to retrieve predictions from memory in video coding applications, particularly for memory word sizes (memory bus widths) larger than the size of typical predictions. According to a second aspect, two or more 2D subarrays such as video predictions are retrieved from memory simultaneously as part of a larger 2D array, if retrieving the larger array requires fewer clock cycles than retrieving the subarrays individually. Allowing the combination of multiple predictions in one memory access operation can lead to a reduction in the average bandwidth required to retrieve predictions from memory.
1. A video coding method comprising:
determining a first number of memory access clock cycles needed to retrieve a first prediction from the random access memory;
determining a second number of memory access clock cycles needed to retrieve a second prediction from the random access memory, the second prediction being distinct from the first prediction;
determining a third number of memory access clock cycles needed to retrieve a combined prediction containing the first prediction and the second prediction from the random access memory;
comparing the third number of clock cycles with a sum of the first number of clock cycles and the second number of clock cycles; and
sending to the random access memory a memory read command to retrieve the combined prediction from the random access memory when the third number of clock cycles is less than or equal to the sum of the first number of clock cycles and the second number of clock cycles;
wherein both the first and second predictions are image blocks that are located in a first image frame and associated with a macroblock in a second image, a first difference in the locations between the first predication and the macroblock being represented by a first motion vector, a second difference in the locations between the second predication and the macroblock being represented by a second motion vector;
wherein each of the first prediction and the second prediction is smaller than or equal to the macroblock in size.
2. The method of claim 1 , further comprising adding the first prediction to a first residual to generate a first target data set, and adding the second prediction to a second residual to generate a second target data set.
3. The method of claim 1 , further comprising performing a first subtraction operation between a first target data set and the first prediction to generate a first residual, and a second subtraction operation between a second target data set and the second prediction to generate a second residual.
4. The method of claim 1 , wherein the first prediction overlaps the second prediction.
5. The method of claim 1 wherein the first prediction does not overlap the second prediction.
6. The method of claim 1 , wherein an area of the combined prediction is defined by a rectangle of minimal area that contains an area of the first prediction and an area of the second prediction, and that contains an integer number of memory words.
7. The method of claim 1 , further comprising evaluating whether to combine the first prediction with each of a plurality of predictions.
8. The method of claim 7 , further comprising evaluating whether to combine the combined prediction with each of at least a subset of the plurality of predictions.
9. The method of claim 1 , further comprising retrieving the combined prediction from the random access memory when the third number of clock cycles is less than or equal to the sum of the first number of clock cycles and the second number of clock cycles.
10. The method of claim 1 , wherein the combined prediction includes an image part present in neither of the first prediction and the second prediction.
11. The method of claim 1 , wherein each of the first prediction and the second prediction has a luminance size of 4×4, 4×8, 8×8, 16×8 or 16×16.
12. A video coding method comprising:
comparing a memory bandwidth requirement for retrieving a combined prediction containing a first prediction and a second prediction from a random access memory with a memory bandwidth requirement for retrieving individually the first prediction and the second prediction from the random access memory; and
retrieving from the random access memory either the combined prediction or the first and second predictions individually according to a comparison of the memory bandwidth requirement for retrieving the combined prediction to the memory bandwidth requirement for retrieving individually the first prediction and the second prediction;
wherein both the first and second predictions are image blocks that are located in a first image frame and associated with a macroblock in a second image, a first difference in the locations between the first predication and the macroblock being represented by a first motion vector, a second difference in the locations between the second predication and the macroblock being represented by a second motion vector;
wherein each of the first prediction and the second prediction is smaller than or equal to the macroblock in size.
13. The method of claim 12 , further comprising performing a video decoding operation using the first prediction and the second prediction.
14. The method of claim 12 , further comprising performing a video encoding operation using the first prediction and the second prediction.
15. The method of claim 12 , wherein the first prediction overlaps the second prediction.
16. The method of claim 12 , wherein the first prediction does not overlap the second prediction.
17. The method of claim 12 , wherein an area of the combined prediction is defined by a rectangle of minimal area that contains an area of the first prediction and an area of the second prediction, and that contains an integer number of memory words.
18. The method of claim 12 , further comprising evaluating whether to combine the first prediction with each of a plurality of predictions.
19. The method of claim 18 , further comprising evaluating whether to combine the combined prediction with each of at least a subset of the plurality of predictions.
20. The method of claim 12 , comprising retrieving the combined prediction when the memory bandwidth requirement for retrieving the combined prediction is not larger than the memory bandwidth requirement for retrieving individually the first prediction.
21. The method of claim 12 , wherein the combined prediction includes an image part present in neither of the first prediction and the second prediction.
22. The method of claim 12 , wherein each of the first prediction and the second prediction has a luminance size of 4×4, 4×8, 8×8, 16×8 or 16 × 16 .
23. A video coding apparatus comprising:
comparison logic for comparing a memory bandwidth requirement for retrieving a combined prediction containing a first prediction and a second prediction from a random access memory with a memory bandwidth requirement for retrieving individually the first prediction and the second prediction from the random access memory; and
memory access logic connected to the comparison logic, for retrieving the combined prediction or the first and second predictions individually according to a result of complaining the memory bandwidth requirement for retrieving the combined prediction to the memory bandwidth requirement for retrieving individually the first prediction and the second prediction;
both the first and second predictions are image blocks that are located in a first image frame and associated with a macroblock in a second image, a first difference in the locations between the first predication and the macroblock being represented by a first motion vector, a second difference in the locations between the second predication and the macroblock being represented by a second motion vector;
wherein each of the first prediction and the second prediction is smaller than or equal to the macroblock in size.
24. The apparatus of claim 23 , wherein the first prediction overlaps the second prediction.
25. The apparatus of claim 23 , wherein the memory access logic is configured to retrieve the combined prediction when the memory bandwidth requirement for retrieving the combined prediction is not larger than the memory bandwidth requirement for retrieving individually the first prediction and the second prediction.
26. The apparatus of claim 23 , wherein the combined prediction includes an image part present in neither of the first prediction and the second prediction.
27. The apparatus of claim 23 , wherein each of the first prediction and the second prediction has a luminance size of 4×4, 4×8, 8×8, 16×8 or 16×16.
28. A video coding apparatus comprising:
means for comparing a memory bandwidth requirement for retrieving a combined prediction containing a first prediction and a second prediction from a random access memory with a memory bandwidth requirement for retrieving individually the first prediction and the second prediction from the random access memory; and
means for retrieving the combined prediction or the first and second predictions individually according to a result of comparing of the memory bandwidth requirement for retrieving the combined prediction to the memory bandwidth requirement for retrieving individually the first prediction and the second prediction;
wherein both the first and second predictions are image blocks that are located in a first image frame and associated with a macroblock in a second image, a first difference in the locations between the first predication and the macroblock being represented by a first motion vector, a second difference in the locations between the second predication and the macroblock being represented by a second motion vector;
wherein each of the first prediction and the second prediction is smaller than or equal to the macroblock in size.
29. The apparatus of claim 28 , wherein the first prediction overlaps the second prediction.
30. The apparatus of claim 28 , wherein the means for retrieving either the combined prediction or the first and second predictions individually is configured to retrieve the combined prediction when the memory bandwidth requirement for retrieving the combined prediction is not larger than the memory bandwidth requirement for retrieving individually the first prediction and the second prediction.
31. The apparatus of claim 28 , wherein the combined prediction includes an image part present in neither of the first prediction and the second prediction.
32. The apparatus of claim 28 , wherein each of the first prediction and the second prediction has a luminance size of 4×4, 4×8, 8×8, 16×8 or 16×16.