Parallel processing of digital images
A system and method for providing video conferencing are described herein. Original video streams from one or more cameras are sampled to form downscaled versions of the video stream. The downscaled version is analyzed to find regions of interest and generate metadata describing aspects of the regions of interest, including a region that includes a preferred speaker and the best view of each participant present together in the video conference. Cropping instructions are generated from the metadata, and portions of the original video stream are removed to form a set of presentation views for each participant, including the preferred speaker. The set of presentation views is stitched together to form a composite display for each participant viewing the stream remotely.
1. A method of video conferencing, comprising:
processing a video stream received from each of a plurality of sensors, wherein each video stream includes a first version of video data having a first resolution, and processing the video stream comprises:
sampling the first version of the video data to form a second version of the video data, wherein the second version of the video data has a second resolution that is less than the first resolution;
determining one or more regions of interest within the second version of the video data, and
generating metadata for each of the one or more regions of interest;
generating cropping instructions for each of the one or more regions of interest based on the metadata;
removing portions of the first version of the video data based on the cropping instructions; and
generating a composite display that includes portions of the first version of the video data remaining after removing portions from the first version of video data.
2. The method of claim 1 , further comprising determining a preferred speaker captured within the video stream based on the metadata.
3. The method of claim 2 , further comprising generating cropping instructions for a region of interest that includes the preferred speaker.
4. The method of claim 3 , further comprising removing portions of the first version of the video data outside the region of interest that includes the preferred speaker.
5. The method of claim 1 , wherein the plurality of sensors comprise:
a first camera that has a field-of-view that is greater than 180 degrees;
a second camera that has a field-of-view that is greater than 180 degrees; and
a third camera that has a field-of-view that is less than 180 degrees.
6. The method of claim 1 , wherein
the metadata includes bounding box instructions, and
the bounding box instructions include information relating to pixel coordinates that define the one or more regions of interest.
7. The method of claim 6 wherein the metadata further includes data for at least one of face size, face angle, vertical motion history, history of speech, and face direction history.
8. A video conferencing system comprising:
a plurality of sensors that are each configured to generate a video stream that comprises a first version of video data that has a first resolution;
an image signal processor that is configured to downscale the first version of the video data to form a second version of the video data that has a second resolution that is less than the first resolution;
a computer vision processor that is configured to determine two or more regions of interest and generate metadata for each of the two or more regions of interest using the second version of the video data;
a virtual cinematographer configured to:
create a ranking of the determined two or more regions of interest based on the metadata;
generate crop instructions for each of the two or more regions of interest based on the metadata and the ranking of each of the one or more regions of interest; and
crop at least two or more portions of the first version of the video data to form at least two or more presentation regions of interest; and
a video composer configured to compile the at least two or more presentation regions of interest.
9. The system of claim 8 , further including a second image signal processor configured to upscale the presentation regions of interest.
10. The system of claim 8 , wherein the plurality of sensors comprise:
a first camera that has a field-of-view that is greater than 180 degrees;
a second camera that has a field-of-view that is greater than 180 degrees; and
a third camera that has a field-of-view that is less than 180 degrees.
11. The system of claim 8 , wherein the two or more presentation regions of interest are based on a ranking of the two or more presentation regions of interest being higher than other regions of interest in the second version of the video data.
12. The system of claim 8 , wherein the metadata includes data for face angle, color histogram, vertical motion history, history of speech, polar coordinate history, and face direction history.
13. The system of claim 8 , wherein the computer vision processor is further configured to perform at least one of the following:
generate a bounding box;
determine a face angle;
generate a color histogram;
generate vertical motion history;
determine a history of speech;
determine polar coordinates;
generate face direction history; and
issue unique identifiers for each video stream.
14. A method of video conferencing, the method comprising:
processing a video stream received from each of a plurality of sensors, wherein each video stream includes a first version of video data having a first resolution, and processing the video stream comprises:
sampling the first version of the video data to form a second version of the video data, wherein the second version of the video data has a second resolution that is less than the first resolution;
determining one or more regions of interest within the second version of the video data;
generating metadata for each of the one or more regions of interest;
selecting one or more regions of interest as best by ranking each of the one or more regions of interest determined in the second version of the video data based on the metadata;
generating cropping instructions for each of the one or more regions of interest selected as best based on the metadata;
cropping at least two or more portions of video data from the first versions of the video to form two or more presentation regions of interest based on the cropping instructions; and
generating a composite scene video data that includes the two or more presentation regions of interest.
15. The method of claim 14 , wherein ranking each of the one or more regions of interest includes assigning a value to a face angle, color histogram, vertical motion history, history of speech, polar coordinate history, and face direction history of each person.
16. The method of claim 14 , wherein the metadata includes bounding box instructions that include information relating to pixel coordinates that define the one or more regions of interest.
17. The method of claim 14 , wherein the two or more presentation regions of interest are based on a ranking of the two or more presentation regions of interest being higher than other regions of interest in the second version of the video data.
18. The method of claim 14 , further comprising determining a preferred speaker based on the metadata, wherein at least one of the two or more presentation regions of interest include the preferred speaker.
19. The method of claim 14 , wherein the plurality of sensors comprise:
a first camera that has a field-of-view that is greater than 180 degrees;
a second camera that has a field-of-view that is greater than 180 degrees; and
a third camera that has a field-of-view that is less than 180 degrees.
20. The method of claim 14 , wherein generating a composite scene video includes stitching together the two or more presentation regions of interest.