Determining which participant is speaking in a videoconference
View Patent ↗Aspects herein describe methods and systems of receiving, by one or more cameras, images in which the images comprise facial images of individuals. Aspects of the disclosure describe extracting the facial images from the images received, sorting the extracted facial images into separate groups wherein each group corresponds to the facial images of each individual, and selecting, for each individual, a preferred facial image from each group. The preferred facial images selected are transmitted to a client for display. Aspects of the disclosure also describe selecting either a facial recognition algorithm or an audio triangulation algorithm to use to determine which individual is speaking wherein the selection is based on whether lip movement of one or more of the individuals is visible in the images received from the cameras.
1. A system comprising:
one or more cameras; and
a device configured to be connected to the one or more cameras, the device comprising one or more processors and memory storing instructions that, when executed by one of the processors, cause the device to:
obtain, from at least one of the cameras, a set of images of a plurality of individuals at a location,
select, from the set of images, for each individual, a preferred facial image for the individual,
determine whether lip movement of one of the individuals is visible in the set of images, and
select, based on whether lip movement of one of the individuals is visible in the set of images, at least one of a facial recognition algorithm and an audio triangulation algorithm to determine which individual is speaking;
wherein the instructions, when executed by one of the processors, further cause the device to provide a graphical window which is rendered on a display, the graphical window including multiple viewports, the graphical window being divided into a grid pattern, each viewport of the multiple viewports residing in a respective location of the grid pattern and including a facial image for an individual, the graphical window further including viewport highlighting which highlights viewports corresponding to individuals that are speaking.
2. The system of claim 1 , wherein:
the viewport highlighting comprises flashing of one or more viewport borders.
3. The system of claim 1 , wherein:
the viewport highlighting comprises shading a background of one or more viewport portions.
4. The system of claim 1 , wherein:
the instructions, when executed by one of the processors, cause the device to:
determine that multiple individuals are speaking simultaneously, and
cause multiple viewports that present facial images of the multiple individuals to provide indications that the multiple individuals are speaking.
5. The system of claim 1 , wherein:
determining whether lip movement of one of the individuals is visible in the set of images comprises determining whether a lip area of the individual is visible in the set of images.
6. The system of claim 1 , wherein:
the set of images comprises video of the individuals participating in a videoconference at the location.
7. A device comprising:
one or more processors; and
memory storing instructions that, when executed by one of the processors, cause the device to:
obtain, from at least one camera, a set of images of a plurality of individuals at a location,
select, from the set of images, for each individual, a preferred facial image for the individual,
determine whether lip movement of one of the individuals is visible in the set of images, and
select, based on whether lip movement of one of the individuals is visible in the set of images, at least one of a facial recognition algorithm and an audio triangulation algorithm to determine which individual is speaking;
wherein the instructions, when executed by one of the processors, further cause the device to provide a graphical window which is rendered on a display, the graphical window including multiple viewports, the graphical window being divided into a grid pattern, each viewport of the multiple viewports residing in a respective location of the grid pattern and including a facial image for an individual, the graphical window further including viewport highlighting which highlights viewports corresponding to individuals that are speaking.
8. The device of claim 7 , wherein:
the viewport highlighting comprises flashing of one or more viewport borders.
9. The device of claim 7 , wherein:
the viewport highlighting comprises shading a background of one or more viewport portions.
10. The device of claim 7 , wherein:
the instructions, when executed by one of the processors, cause the device to:
determine that multiple individuals are speaking simultaneously, and
cause multiple viewports that present facial images of the multiple individuals to provide indications that the multiple individuals are speaking.
11. The device of claim 7 , wherein:
determining whether lip movement of one of the individuals is visible in the set of images comprises determining whether a lip area of the individual is visible in the set of images.
12. The device of claim 7 , wherein:
the set of images comprises video of the individuals participating in a videoconference at the location.
13. A non-transitory computer-readable medium storing instructions, that when executed by a processor of a device, cause the device to:
obtain, from at least one camera, video of a plurality of individuals participating in a video conference,
select, from the video, for each individual, a preferred facial image for the individual,
determine whether lip movement of one of the individuals is visible in the video, and
select, based on whether lip movement of one of the individuals is visible in the video, at least one of a facial recognition algorithm and an audio triangulation algorithm to determine which individual is speaking;
wherein the instructions, when executed, further cause the device to provide a graphical window which is rendered on a display, the graphical window including multiple viewports, the graphical window being divided into a grid pattern, each viewport of the multiple viewports residing in a respective location of the grid pattern and including a facial image for an individual, the graphical window further including viewport highlighting which highlights viewports corresponding to individuals that are speaking.
14. The computer-readable medium of claim 13 , wherein:
the viewport highlighting comprises shading a background of one or more viewport portions.
15. The computer-readable medium of claim 13 , wherein:
determining whether lip movement of one of the individuals is visible in the video comprises determining whether a lip area of the individual is visible in the video.
16. A method of managing a videoconference which includes multiple participants, the method comprising:
for each participant of the videoconference, selecting a preferred video feed that includes that participant from a respective set of video feeds that include that participant;
for each participant of the videoconference, extracting a series of facial images of that participant from the selected preferred video feed that includes that participant; and
provide a graphical window which is rendered on a display, the graphical window including multiple viewports, the graphical window being divided into a grid pattern, each viewport of the multiple viewports residing in a respective location of the grid pattern and including one of the series of facial images extracted for a particular participant, the graphical window further including viewport highlighting which highlights viewports corresponding to participants that are speaking.
17. A method as in claim 16 , further comprising:
identifying exactly one participant that is speaking based on (i) detected lip movement and (ii) audio triangulation; and
wherein the viewport highlighting highlights exactly one viewport which corresponds to the exactly one participant that is speaking.
18. A method as in claim 16 , further comprising:
based on (i) detected lip movement and (ii) audio triangulation, identifying a first participant and a second participant that are simultaneously speaking; and
wherein the viewport highlighting concurrently highlights (i) a first viewport which corresponds to the first participant and (ii) a second viewport which corresponds to the second participant, the first viewport and the second viewport residing in different locations of the grid pattern.