Optimal view selection method in a video conference
A system for ensuring that the best available view of a person's face is included in a video stream when the person's face is being captured by multiple cameras at multiple angles at a first endpoint. The system uses one or more microphone arrays to capture direct-reverberant ratio information corresponding to the views, and determines which view most closely matches a view of the person looking directly at the camera, thereby improving the experience for viewers at a second endpoint.
1. A method for determining an optimal camera view from amongst a plurality of views, the method comprising:
locating an active talker at a first endpoint by a first electronic device using a first sound source location module;
deriving second-order information from the first sound source location module;
locating the active talker at the first endpoint by a second electronic device using a second sound source location module;
deriving second-order information from the second sound source location module;
capturing a first face view of the active talker using a camera of the first electronic device;
capturing a second face view of the active talker using a camera of the second electronic device;
detecting a low-frequency signal energy (LFE) corresponding to the first face view and detecting a LFE corresponding to the second face view;
determining whether a disparity between the LFE corresponding to the first face view and that of the second face view exceeds a predetermined threshold;
including, in event the disparity exceeds the threshold, the LFE of each device within the second-order information;
determining an optimal view from amongst the first face view and second face view, based on the second-order information; and
including the optimal view in a video stream for transmission to a second endpoint.
2. The method of claim 1 , wherein the second-order information for each device includes pan-ratio (PR), number-of-counts (NC) within a predetermined period, and a ratio of high-band energy signals to low-band energy signals (HBLD).
3. The method of claim 2 , wherein the predetermined period is two seconds.
4. The method of claim 1 , wherein determining an optimal view from amongst the first face view and second face view, based on the second-order information, comprises determining a second-order-information score (SOIC) for the first electronic device and a SOIC for the second electronic device, and selecting the face view corresponding to the greater SOIC.
5. The method of claim 1 , wherein determining whether a disparity between the LFE corresponding to the first face view and that of the second face view exceeds a predetermined threshold comprises determining whether the greater LFE exceeds the lesser LFE by a factor of two.
6. A videoconferencing system for determining an optimal camera view from amongst a plurality of views, the videoconferencing system configured to:
locate an active talker at a first endpoint by a first electronic device using a first sound source location module;
derive second-order information from the first sound source location module;
locate the active talker at the first endpoint by a second electronic device using a second sound source module;
derive second-order information from the sound source location module;
capture a first face view of the active talker using a camera of the first electronic device;
capture a second face view of the active talker using a camera of the second electronic device;
detect a low-frequency signal energy (LFE) corresponding to the first face view and detecting a LFE corresponding to the second face view;
determine whether a disparity between the LFE corresponding to the first face view and that of the second face view exceeds a predetermined threshold;
include, in event the disparity exceeds the threshold, the LFE of each device within the second-order information;
determine an optimal view from amongst the first face view and second face view, based on the second-order information, wherein the second-order information for each device includes pan-ratio and a number of times that the sound source location module was able to determine a source of sound corresponding to the active talker within a predetermined period; and
include the optimal view in a video stream for transmission to a second endpoint.
7. The videoconferencing system of claim 6 , wherein the predetermined period is two seconds.
8. The videoconferencing system of claim 6 , wherein determining an optimal view from amongst the first face view and second face view, based on the second-order information, comprises determining a second-order-information score (SOIC) for the first electronic device and a SOIC for the second electronic device, and selecting the face view corresponding to the greater SOIC.
9. The videoconferencing system of claim 6 , wherein determining whether a disparity between the LFE corresponding to the first face view and that of the second face view exceeds a predetermined threshold comprises determining whether the greater LFE exceeds the lesser LFE by a factor of two.
10. A non-transitory computer readable storage medium storing instructions, the instructions executable by a processor to cause the processor to:
locate an active talker at a first endpoint by a first electronic device using a first sound source location module;
derive second-order information from the first sound source location module;
locate the active talker at the first endpoint by a second electronic device using a second sound source module;
derive second-order information from the sound source location module;
capture a first face view of the active talker using a camera of the first electronic device;
capture a second face view of the active talker using a camera of the second electronic device;
detect a low-frequency signal energy (LFE) corresponding to the first face view and detecting a LFE corresponding to the second face view;
determine whether a disparity between the LFE corresponding to the first face view and that of the second view exceeds a predetermined threshold;
include the LFE of each device within the second-order information when a disparity between the LFE corresponding to the first face view and the LFE of the second face view exceeds a predetermined threshold;
determine an optimal view from amongst the first face view and second face view, based on the second-order information; and
include the optimal view in a video stream for transmission to a second endpoint.
11. The non-transitory computer readable storage medium of claim 10 , wherein the second-order information for each device includes pan-ratio (PR), number-of-counts (NC) within a predetermined period, and a ratio of high-band energy signals to low-band energy signals (HBLD).
12. The non-transitory computer readable storage medium of claim 11 , wherein the predetermined period is two seconds.
13. The non-transitory computer readable storage medium of claim 10 , wherein determining the optimal view from amongst the first face view and second face view, based on the second-order information, comprises determining a second-order-information score (SOIC) for the first electronic device and a SOIC for the second electronic device, and selecting the face view corresponding to the greater SOIC.
14. The non-transitory computer readable storage medium of claim 10 , wherein determining whether a disparity between the LFE corresponding to the first face view and that of the second face view exceeds a predetermined threshold comprises determining whether the greater LFE exceeds the lesser LFE by a factor of two.
15. The method of claim 1 , further comprising switching from a current view to the optimal view.
16. The method of claim 1 , wherein the optimal view comprises a front face view of the active talker.
17. The videoconferencing system of claim 6 , wherein the videoconferencing system is further configured switch from a current view to the optimal view.
18. The videoconferencing system of claim 6 , wherein the optimal view comprises a front face view of the active talker.
19. The non-transitory computer readable storage medium of claim 10 , wherein the optimal view comprises a front face view of the active talker.