Real-time video frame pre-processing hardware
View Patent ↗A dynamically reconfigurable heterogeneous systolic array is configured to process a first image frame, and to generate image processing primatives from the image frame, and to store the primatives and the corresponding image frame in a memory store. A characteristic of the image frame is determined. Based on the characteristic, the array is reconfigured to process a following image frame.
1. An image processing hardware unit comprising:
a frame receiving component configured to:
receive, in a dynamically reconfigurable heterogeneous systolic array, frame data of a first image frame;
a frame processing component configured to:
in the dynamically reconfigurable heterogeneous systolic array, generate, based on the frame data of the first image frame, an integral image primitive;
wherein the integral image primitive is generated by identifying a sub-region of the first image frame and generating the integral image primitive from data of the sub-region by computing, for a pixel of the integral image primitive a sum of luminance values of all pixels located above and to the left from the pixel in the sub-region of the first image frame;
store the integral image primitive and the first image frame in a memory store;
based, at least in part, on the integral image primitive, determine at least one characteristic;
based, at least in part, on the at least one characteristic, reconfigure the dynamically reconfigurable heterogeneous systolic array to process, based at least in part on the integral image primitive, a second image frame.
2. The image processing hardware unit of claim 1 , further comprising an integral image accumulator, wherein the integral image accumulator is further configured to:
generate, based on the integral image primitive, an integral image square (primitive as a product of the integral image primitive multiplied by the integral image primitive; and
store the integral image square primitive in the memory store.
3. The image processing hardware unit of claim 2 , wherein the integral image accumulator is further configured to:
for each pixel of the sub-region, determine a similarity score indicating a similarity between a color of the pixel and a skin color;
determine a skin integral image primitive based on the similarity scores determined for the sub-region; and
store the skin integral image primitive in the memory store.
4. The image processing hardware unit of claim 3 ,
wherein output of the integral image primitive is a set of data values corresponding to a feature vector set which is used to determine whether a face is depicted in the sub-region;
wherein the integral image square primitive is used to determine a variance in color values within the sub-region; and
wherein the skin integral image primitive is used to determine a probability that a skin region is depicted in the sub-region.
5. The image processing hardware unit of claim 4 , wherein the frame processing component is further configured to:
determine whether the variance is low, and if so, determine that the sub-region unlikely depicts the face; and
determine whether the variance is high, and if so, determine that the sub-region probably depicts the face.
6. The image processing hardware unit of claim 5 , wherein the first image frame is a video frame.
7. The image processing hardware unit of claim 6 , wherein the frame processing component is further configured to:
generate, based on the frame data of the first image frame, a skin-map primitive;
wherein the skin-map primitive is a bit-map; and
store the skin-map primitive in the memory store.
8. An image processing method comprising:
receiving frame data of a first image frame;
generating, based on the frame data of the first image frame, an integral image primitive;
wherein the image processing primitive is generated by identifying a sub-region of the first image frame and generating the integral image primitive from data of the sub-region by computing for a pixel of the integral image primitive a sum of luminance values of all pixels located above and to the left from the pixel in the sub-region of the first image frame;
storing the integral image primitive and the first image frame in a memory store;
based, at least in part, on the integral image primitive, determining at least one characteristic,
based, at least in part, on the at least one characteristic, reconfiguring a dynamically reconfigurable heterogeneous systolic array to process, based at least in part on the integral image primitive, a second image frame;
wherein the method is performed using one or more computing devices.
9. The image processing method of claim 8 , further comprising:
generating, based on the integral image primitive, an integral image square primitive as a product of the integral image primitive multiplied by the integral image primitive; and
storing the integral image square primitive in the memory store.
10. The image processing method of claim 9 , further comprising:
for each pixel of the sub-region, determining a similarity score indicating a similarity between a color of the pixel and a skin color;
determining a skin integral image primitive based on the similarity scores determined for the sub-region; and
storing the skin integral image primitive in the memory store.
11. The image processing method of claim 10 ,
wherein output of the integral image primitive is a set of data values corresponding to a feature vector set which is used to determine whether a face is depicted in the sub-region;
wherein the integral image square primitive is used to determine a variance in color values within the sub-region; and
wherein the skin integral image primitive is used to determine a probability that a skin region is depicted in the sub-region.
12. The image processing method of claim 11 , further comprising:
determining whether the variance is low, and if so, determine that the sub-region unlikely depicts the face; and
determining whether the variance is high, and if so, determine that the sub-region probably depicts the face.
13. The image processing method of claim 12 , wherein the first image frame is a video frame.
14. The image processing method of claim 13 , further comprising:
generating, based on the frame data of the first image frame, a skin-map primitive;
wherein the skin-map primitive is a bit-map; and
storing the skin-map primitive in the memory store.
15. A non-transitory computer-readable storage medium storing one or more computer instructions which, when executed, cause one or more processors to perform:
receiving frame data of a first image frame;
generating, based on the frame data of the first image frame, an integral image primitive;
wherein the integral image primitive is generated by identifying a sub-region of the first image frame and generating the integral image primitive from data of the sub-region by computing for a pixel of the integral image primitive a sum of luminance values of all pixels located above and to the left from the pixel in the first image frame; and
storing the integral image primitive and the first image frame in a memory store;
based, at least in part, on the integral image primitive, determine at least one characteristic;
based, at last in part, on the at least one characteristic, reconfiguring a dynamically reconfigurable heterogeneous systolic array to process, based at least in part on the integral image primitive, a second image frame.
16. The non-transitory computer-readable storage medium of claim 15 , further comprising additional instructions which, when executed, cause the one or more processors to perform:
generating, based on the integral image primitive, an integral image square primitive as a product of the integral image primitive multiplied by the integral image primitive; and
storing the integral image primitive in the memory store.
17. The non-transitory computer-readable storage medium of claim 16 , further comprising additional instructions which, when executed, cause the one or more processors to perform:
for each pixel of the sub-region, determining a similarity score indicating a similarity between a color of the pixel and a skin color;
determining a skin integral image primitive based on the similarity scores determined for the sub-region; and
storing the skin integral image primitive in the memory store.
18. The non-transitory computer-readable storage medium of claim 17 ,
wherein output of the integral image primitive is a set of data values corresponding to a feature vector set which is used to determine whether a face is depicted in the sub-region;
wherein the integral image square primitive is used to determine a variance in color values within the sub-region; and
wherein the skin integral image primitive is used to determine a probability that a skin region is depicted in the sub-region.
19. The non-transitory computer-readable storage medium of claim 18 , further comprising additional instructions which, when executed, cause the one or more processors to perform:
determining whether the variance is low, and if so, determine that the sub-region unlikely depicts the face; and
determining whether the variance is high, and if so, determine that the sub-region probably depicts the face.
20. The non-transitory computer-readable storage medium of claim 19 , wherein the first image frame is a video frame.