Optimal view selection method in a video conference
A system for ensuring that the best available view of a person's face is included in a video stream when the person's face is being captured by multiple cameras at multiple angles at a first endpoint. The system uses one or more microphone arrays to capture direct-reverberant ratio information corresponding to the views, and determines which view most closely matches a view of the person looking directly at the camera, thereby improving the experience for viewers at a second endpoint.
1. A method for determining an optimal camera view from amongst a plurality of views, the method comprising:
ascertaining an active talker using a first sound location module, capturing a first view of the active talker using a first camera unit, and deriving a first set of second-order information corresponding to the first view;
ascertaining the active talker using a second sound location module, capturing a second view of the active talker using a second camera, and deriving a second set of second-order information corresponding to the second view;
detecting low-frequency energy (LFE) corresponding to the first and second views;
determining that the LFE corresponding to the first view differs from the LFE corresponding to the second view in excess of a predetermined value;
including, based on the determination, the LFE corresponding to the first view within the first set of second-order information, and the LFE corresponding to the second view within the second set of second-order information;
determining an optimal view from amongst the first and second views based, at least in part, on the first set of second-order information and the second set of second-order information; and
including the optimal view in a video stream.
2. The method of claim 1 , wherein including the optimal view in the video stream comprises switching a current view in the video stream to the optimal view.
3. The method of claim 1 , wherein the first sound location module is different from the second sound location module.
4. The method of claim 1 , wherein the first set of second-order information includes first pan-ratio information, and the second set of second-order information includes second pan-ratio information.
5. The method of claim 1 , wherein the first set of second-order information includes first number-of-counts (NC) information and the second set of second-order information includes second number-of-counts information.
6. The method of claim 1 , wherein the first sound location module is coupled to a first microphone array, and the second sound location module is coupled to a second microphone array.
7. The method of claim 6 , wherein the first microphone array is different from the second microphone array.
8. A non-transitory computer readable medium storing instructions executable by one or more processors, the instructions comprising instructions to:
ascertain an active talker using a first sound location module, capture a first view of the active talker using a first camera unit, and derive a first set of second-order information corresponding to the first view;
ascertain the active taker using a second sound location module, capturing a second view of the active taker using a second camera, and derive a second set of second-order information corresponding to the second view;
detect low-frequency energy (LFE) corresponding to the first and second views;
determine that the LFE corresponding to the first view differs from the LFE corresponding to the second view in excess of a predetermined value;
include, based on the determination, the LFE corresponding to the first view within the first set of second-order information, and the LFE corresponding to the second view within the second set of second-order information;
determine an optimal view from amongst the first and second views based, at least in part, on the first set of second-order information and the second set of second-order information; and
include the optimal view in a video stream.
9. The non-transitory computer readable medium of claim 8 , wherein the instructions to include the optimal view in a video stream comprise instructions to replace a current view in the video stream with the optimal view.
10. The non-transitory computer readable medium of claim 9 , wherein the instructions to include the optimal view in a video stream comprise instructions to determine that the current view has been included in the video stream for a time period which exceeds a predetermined threshold.
11. The non-transitory computer readable medium of claim 8 , wherein the first sound location module is different from the second sound location module.
12. The non-transitory computer readable medium of claim 8 , wherein the first set of second-order information includes first pan-ratio information, and the second set of second-order information includes second pan-ratio information.
13. The non-transitory computer readable medium of claim 8 , wherein the first set of second-order information includes first number-of-counts information and the second set of second-order information includes second number-of-counts information.
14. The non-transitory computer readable medium of claim 9 , wherein the first sound location module is coupled to a first microphone array, and the second sound location module is coupled to a second microphone array.
15. The non-transitory computer readable medium of claim 14 , wherein the first microphone array is different from the second microphone array.
16. A teleconferencing endpoint comprising:
one or more processors;
a first camera unit and a first microphone array, the first camera unit and the first microphone array coupled to the one or more processors;
a second camera unit and second microphone array, the second camera unit and the second microphone array coupled to the one or more processors; and
a non-transitory computer readable medium storing instructions executable by the one or more processors, the instructions comprising instructions to:
capture first audio using the first microphone array, ascertain an active talker using a first sound location module based on the first audio, capture a first view of the active talker using the first camera unit, and derive a first set of second-order information corresponding to the first view;
capture second audio using the second microphone array, ascertain the active talker using a second sound location module based on second audio, capture a second view of the active talker using the second camera unit, and derive a second set of second-order information corresponding to the second view;
detect low-frequency energy (LFE) corresponding to the first audio and the second audio;
determine that the LFE corresponding to the first audio differs from the LFE corresponding to the second audio in excess of a predetermined value;
include, based on the determination, the LFE corresponding to the first audio within the first set of second-order information, and the LFE corresponding to the second audio within the second set of second-order information;
determine an optimal view from amongst the first and second views based, at least in part, on the first set of second-order information and the second set of second-order information; and
include the optimal view in a video stream.
17. The teleconferencing endpoint of claim 16 , wherein the instructions to include the optimal view in a video stream comprise instructions to replace a current view in the video stream with the optimal view.
18. The teleconferencing endpoint of claim 16 , wherein the first sound location module is different from the second sound location module.
19. The teleconferencing endpoint of claim 16 , wherein the first set of second-order information includes first pan-ratio information, and the second set of second-order information includes second pan-ratio information.
20. The teleconferencing endpoint of claim 16 , wherein the first set of second-order information includes first number-of-counts information and the second set of second-order information includes second number-of-counts information.