Advanced video conferencing systems and methods
The present disclosure generally provides for advanced single camera video conferencing systems, and methods related thereto. The advanced single camera video conferencing system features a hybrid optical/digital camera, herein a camera device, having a controller that is configured to execute one or more of the methods set forth herein. In one embodiment, a method includes optically framing, a first portion of a video conferencing environment to provide an actual field-of-view, digitally framing a second portion of the video conferencing environment to provide an apparent field-of-view that is encompassed within the actual field-of-view, generating a video stream of the apparent field-of-view, surveying the actual field-of-view to generate survey data, and detecting changes in the survey data over time. The method may be performed using a single camera device using a single image sensor.
1. A computer implemented method, comprising:
generating, by use of a camera device, a video stream that comprises an apparent field-of-view of a physical environment;
generating, by use of the camera device, one or more survey frames of the physical environment, wherein each of the one or more survey frames comprise at least a portion of an actual field-of-view of the camera device; and
analyzing the one or more survey frames to generate survey data,
wherein generating the video stream further comprises:
analyzing the apparent field-of-view to determine an actual composition thereof;
comparing the actual composition of the apparent field-of-view to one or more composition rules;
analyzing the survey data, using the one or more composition rules, to determine a desired composition of the apparent field-of-view; and
altering the apparent field-of-view based on a difference between the actual composition and the desired composition, wherein the apparent field-of-view is altered after comparing the difference between the actual composition and the desired composition with an anti-movement rule and a determination is made, based on the comparison of the difference to then anti-movement rule, that the alteration to the apparent field-of-view should be made.
2. The method of claim 1 , wherein the apparent field-of-view is a portion of the actual field-of-view.
3. The method of claim 1 , wherein the difference between the actual composition and the desired composition is based on a change in location of one or more objects within the physical environment between one or more generated survey frames.
4. The method of claim 3 , wherein the physical environment is a video conferencing environment and the one or more objects are one or more conference participants.
5. The method of claim 3 , wherein the one or more composition rules are based on one or a combination of negative space surrounding the one or more objects, negative space between the one or more objects and a boundary of the apparent field-of-view, asymmetry of the location of the one or more objects within the apparent field-of-view, a direction one or more objects are facing within the apparent field-of-view, a depth of field between at least two objects, and a ratio of a depth to width of the physical environment relative to the camera device.
6. The method of claim 1 , altering the apparent field-of-view comprises one or a combination of panning, tilting, optically zooming, and digitally zooming the camera device.
7. A computer implemented method, comprising:
generating, by use of a camera device, a video stream comprising first data acquired from an image sensor, wherein the first data corresponds only to an apparent field-of-view of a physical environment and is acquired without performing a post-processing process to remove data acquired from the image sensor;
generating, by use of the camera device, one or more survey frames of the physical environment, wherein each of the one or more survey frames comprise second data acquired without performing post-processing process to remove data acquired from the image sensor, wherein the second data corresponds to at least a portion of an actual field-of-view of the camera device, wherein the one or more survey frames are generated between sequentially generated frames of the video stream of the apparent field-of-view, and wherein the method further comprises:
transmitting the video stream to a user device;
extracting one of the one or more survey frames from the video stream before the video stream is transmitted to the user device; and
analyzing the one or more survey frames to generate survey data,
wherein generating the video stream further comprises:
analyzing the apparent field-of-view to determine an actual composition thereof;
comparing the actual composition of the apparent field-of-view to one or more composition rules;
analyzing the survey data, using the one or more composition rules, to determine a desired composition of the apparent field-of-view; and
altering the apparent field-of-view based on a difference between the actual composition and the desired composition.
8. The method of claim 7 , wherein a video conferencing software application which is being executed on the user device is configured to receive the video stream and transmit the video stream to a remote video conferencing device for display.
9. A computer implemented method, comprising:
generating, by use of a camera device, a video stream that comprises an apparent field-of-view of a physical environment;
generating, by use of the camera device, one or more survey frames of the physical environment, wherein each of the one or more survey frames comprise at least a portion of an actual field-of-view of the camera device; and
analyzing the one or more survey frames to generate survey data,
wherein generating the video stream further comprises:
analyzing the apparent field-of-view to determine an actual composition thereof;
comparing the actual composition of the apparent field-of-view to one or more composition rules;
analyzing the survey data, using the one or more composition rules, to determine a desired composition of the apparent field-of-view;
altering the apparent field-of-view based on a difference between the actual composition and the desired composition;
generating a crop-failure score based on the comparison of the actual composition of the apparent field-of-view to the one or more composition rules; and
comparing an accumulated crop-failure score to a crop-failure tolerance limit, wherein the accumulated crop-failure score comprises a plurality of generated crop-failure scores, and wherein the apparent field-of-view is not altered until the accumulated crop-failure score is equal to or outside of the crop-failure tolerance limit.
10. The method of claim 9 , wherein the generating the crop-failure score comprises:
comparing a location of one or more participants relative to one or more boundaries of the apparent field-of-view to a plurality of predetermined crop-failure events each having a corresponding predetermined crop-failure score; and
generating one or more crop-failure scores for the respective locations of at least one of the one or more participants based on the comparison to the plurality of predetermined crop-failure events.
11. The method of claim 9 , wherein
the accumulated crop-failure score comprises a plurality of generated crop-failure scores taken within a rolling time window, and
individual crop-failure scores acquired outside of the rolling time window are not included in the determination of the accumulated crop-failure score.
12. A computer implemented method, comprising:
generating, by use of an image sensor of a camera device, a video stream that comprises an apparent field-of-view of a physical environment;
generating, by use the image sensor of the camera device, one or more survey frames of the physical environment, wherein each of the one or more survey frames comprise at least a portion of an actual field-of-view of the camera device; and
analyzing the one or more survey frames to generate survey data;
wherein generating the video stream further comprises:
analyzing the apparent field-of-view to determine an actual composition thereof;
comparing the actual composition of the apparent field-of-view to one or more composition rules;
analyzing the survey data, using the one or more composition rules, to determine a desired composition of the apparent field-of-view; and
altering the apparent field-of-view based on a difference between the actual composition and the desired composition, wherein altering the apparent field-of-view comprises one or a combination of panning, tilting, optically zooming, and digitally zooming the camera device, and wherein altering the apparent field-of-view also alters the actual field-of-view.
13. A video conferencing method, comprising:
defining, by use of a camera device, an actual field-of-view, wherein the actual field-of-view optically frames a portion of a video conferencing environment;
generating, by use of the camera device, one or more survey frames that each comprise at least a portion of the actual field-of-view;
analyzing the one or more survey frames to generate survey data;
generating a crop-failure score using the survey data, wherein
the crop-failure score is based on a location of one or more participants relative to one or more boundaries of an apparent field-of-view contained within a first video stream generated by the camera device,
the apparent field-of-view is different from the actual field-of-view; and
altering the apparent field-of-view of the video stream when an accumulated crop-failure score is equal to or outside of a crop-failure tolerance limit, wherein
the accumulated crop-failure score comprises a plurality of generated crop-failure scores taken within a rolling time window, and
individual crop-failure scores acquired outside of the rolling time window are not included in the determination of the accumulated crop-failure score.
14. The method of claim 13 , wherein the apparent field-of-view is defined by digitally framing, by use of the camera device, a second portion of the video conferencing environment, and wherein the apparent field-of-view is a portion of the actual field-of-view.
15. The method of claim 13 , wherein the first video stream comprises a plurality of frames that comprise the apparent field-of-view and the one or more survey frames, and the method further comprises:
generating, by use of the camera device, a to-be-transmitted video stream that comprises the frames that comprise the apparent field-of-view, wherein the frames that comprise the apparent field-of-view comprises data acquired from an image sensor, wherein the data acquired from the image sensor corresponds only to the apparent field-of-view and is acquired without performing a post-processing process to remove data acquired from the image sensor; and
extracting the one or more survey frames from the first video stream so that the to-be-transmitted video stream does not include the one or more survey frames.
16. The method of claim 13 , wherein generating the crop-failure score comprises comparing an actual composition of the apparent field-of-view to one or more composition rules.
17. The method of claim 16 , wherein the one or more composition rules are based on one or a combination of negative space surrounding the one or more participants, negative space between the one or more participants and a boundary of the apparent field-of-view, asymmetry of the location of the one or more participants within the apparent field-of-view, a direction one or more participants are facing within the apparent field-of-view, a depth of field between at least two participants, and a ratio of a depth to width of the video conferencing environment relative to the camera device.
18. A video conferencing method, comprising:
defining, by use of a camera device, an actual field-of-view, wherein the actual field-of-view optically frames a portion of a video conferencing environment;
generating, by use of the camera device, one or more survey frames that each comprise at least a portion of the actual field-of-view;
analyzing the one or more survey frames to generate survey data;
generating a crop-failure score using the survey data, wherein the crop-failure score is based on a location of one or more participants relative to one or more boundaries of an apparent field-of-view contained within a first video stream generated by the camera device, and wherein the apparent field-of-view is different from the actual field-of-view, and generating the crop-failure score comprises comparing an actual composition of the apparent field-of-view to one or more composition rules;
altering the apparent field-of-view of the video stream when the crop-failure score or an accumulated crop-failure score is equal to or outside of a crop-failure tolerance limit; and
analyzing the survey data, using the one or more composition rules, to determine a desired composition of the apparent field-of-view, wherein the apparent field-of-view is altered after comparing a difference between the actual composition and the desired composition with an anti-movement rule and a determination is made based on the comparison that the alteration to the apparent field-of-view should be made.
19. A computer implemented method, comprising:
generating, by use of a camera device, a video stream that comprises an apparent field-of-view of a physical environment;
generating, by use of the camera device, one or more survey frames of the physical environment, wherein the one or more survey frames comprises at least a portion of an actual field-of-view detected by the camera device; and
analyzing the one or more survey frames to generate survey data,
wherein generating the video stream comprises:
analyzing the apparent field-of-view to determine an actual composition thereof;
analyzing the one or more survey frames or the survey data to determine a desired composition of the apparent field-of-view;
determining a difference between one or more attributes of the desired composition of the apparent field-of-view and one or more attributes of the actual composition of the apparent field-of-view;
comparing the determined difference to an anti-movement rule;
determining that an alteration to the apparent field-of-view of view should be made based on the comparison of the difference between the one or more attributes of the desired composition and the one or more attributes of the actual composition to the anti-movement rule; and
altering the apparent field-of-view based on the determination that the alteration should be made, wherein altering the apparent field-of-view comprises changing one or more boundaries of the apparent field-of-view to provide the desired composition.
20. The method of claim 19 , wherein individual ones or more of the one or more survey frames are generated between sequentially generated frames of the apparent field-of-view, and wherein the method further comprises:
transmitting the video stream to a user device; and
extracting the individual ones or more of the one or more survey frames from the video stream before the video stream is transmitted to the user device.
21. The method of claim 19 , wherein the difference between the actual composition and the desired composition is based on a change in location of one or more objects within the physical environment between one or more generated survey frames.
22. The method of claim 21 , wherein the physical environment is a video conferencing environment and the one or more objects are one or more conference participants.
23. The method of claim 21 , wherein the desired composition is determined using one or more composition rules that are based on one or a combination of negative space surrounding the one or more objects, negative space between the one or more objects and a boundary of the apparent field-of-view, asymmetry of the location of the one or more objects within the apparent field-of-view, a direction one or more objects are facing within the apparent field-of-view, a depth of field between at least two objects, and a ratio of a depth to width of the physical environment relative to the camera device.
24. The method of claim 19 , wherein generating the video stream further comprises:
generating a crop-failure score using the survey data; and
comparing the crop-failure score or an accumulated crop-failure score to a crop-failure tolerance limit, wherein the accumulated crop-failure score comprises a plurality of generated crop-failure scores, and wherein the apparent field-of-view is not altered until the crop-failure score or the accumulated crop-failure score is equal to or outside of the crop-failure tolerance limit.
25. The method of claim 24 , wherein generating the crop-failure score comprises:
comparing a location of one or more participants relative to one or more boundaries of the apparent field-of-view to a plurality of predetermined crop-failure events each having a corresponding predetermined crop-failure score; and
generating one or more crop-failure scores for the respective locations of at least one or more participants based on the comparison to the plurality of predetermined crop-failure events.
26. The method of claim 24 , wherein
the accumulated crop-failure score comprises a plurality of generated crop-failure scores taken within a rolling time window, and
individual crop-failure scores acquired outside of the rolling time window are not included in the determination of the accumulated crop-failure score.