Context based target framing in a teleconferencing environment

1. A method of selecting sub-frames of video information for rendering in a teleconferencing system, comprising:

receiving, from a camera device, a video data frame;

detecting a presence, within the video data frame, of data indicative of a face;

designating a portion of the video data frame as a target region based on the data indicative of the face;

receiving, from a microphone device, an audio data frame, the audio data frame associated in time with the video data frame, the audio data frame indicative of a sound source location;

updating an audio-visual map, the audio-visual map corresponding to a plurality of earlier video data frames and audio data frames associated therewith, wherein updating the audio-visual map comprises:

determining that the target region corresponds to a first target of the audio-visual map, the first target corresponding to a meeting participant,

incrementing a facial weight value of the first target, responsive to determining that the target region corresponds to the first target,

determining that the sound source location corresponds to the first target, and

incrementing a first talker weight value of the first target, responsive to determining that the sound source location corresponds to the first target,

comparing the first talker weight value of the first target to a second talker weight value of a second target of the audio-visual map, the second target corresponding to a second participant, and

incrementing a first conversational weight value of the first target and a second conversational weight value of the second target when the first talker weight value of the first target and second talker weight value of the second target each exceed a first threshold and an absolute value of a difference between the first talker weight value of the first target and the second talker weight value of the second target does not exceed a second threshold,

selecting one or more sub-frames of the video data frame, wherein selecting one or more subframes comprises:

selecting a first sub-frame depicting the first target when the facial weight value exceeds a third threshold,

selecting a second sub-frame depicting the first target when the first talker weight value exceeds a fourth threshold, and

selecting a third sub-frame depicting the first target and the second target when the first conversational weight value of the first target and the second conversational weight value of the second target each exceed a fifth threshold; and

including the first sub-frame, the second sub-frame, and the third sub-frame in an audio-video stream for transmission to a remote endpoint.

2. The method of claim 1 , wherein the first sub-frame and the second sub-frame are different.

3. The method of claim 1 , wherein the first sub-frame encompasses the second sub-frame.

4. The method of claim 1 , wherein:

updating the audio-visual map further comprises decrementing a third talker weight value of a third target of the audio-visual map, the third target corresponding to a third participant, responsive to determining that the sound source location corresponds to the first target; and

selecting the first sub-frame comprises selecting the first sub-frame to include a depiction of the third target.

5. The method of claim 1 , further comprising generating the audio-visual map using a convolutional neural network.

6. A non-transitory computer readable medium storing instructions executable by a processor, wherein the instructions comprise instructions to:

receive, from a camera device, a video data frame;

detect a presence, within the video data frame, of data indicative of a face;

designate a portion of the video data frame as a target region based on the data indicative of the face;

receive, from a microphone device, an audio data frame, the audio data frame associated in time with the video data frame, the audio data frame indicative of a sound source location;

update an audio-visual map, the audio-visual map corresponding to a plurality of earlier video data frames and audio data frames associated therewith, wherein the instructions to update the audio-visual map comprise instructions to:

determine that the target region corresponds to a first target of the audio-visual map, the first target corresponding to a meeting participant,

increment a facial weight value of the first target, responsive to determining that the target region corresponds to the first target,

determine that the sound source location corresponds to the first target, and

increment a first talker weight value of the first target, responsive to determining that the sound source location corresponds to the first target;

compare the first talker weight value of the first target to a second talker weight value of a second target of the audio-visual map, the second target corresponding to a second participant;

increment a first conversational weight value of the first target and a second conversational weight value of the second target when the first talker weight value of the first target and second talker weight value of the second target each exceed a first threshold and an absolute value of a difference between the first talker weight value of the first target and the second talker weight value of the second target does not exceed a second threshold;

select one or more sub-frames of the video data frame, wherein the instructions to select one or more subframes comprise instructions to:

select a first sub-frame depicting the first target when the facial weight value exceeds a third threshold,

select a second sub-frame depicting the first target when the first talker weight value exceeds a fourth threshold, and

select a third sub-frame depicting the first target and the second target when the first conversational weight value of the first target and the second conversational weight value of the second target each exceed a fifth threshold; and

render, one or more selected sub-frames using at least one display device.

7. The non-transitory computer readable medium of claim 6 , wherein the first sub-frame and the second sub-frame are different.

8. The non-transitory computer readable medium of claim 6 , wherein the first sub-frame and the second sub-frame partially intersect.

9. The non-transitory computer readable medium of claim 6 , wherein:

the instructions to update the audio-visual map further comprise instructions to decrement a third talker weight value of a third target of the audio-visual map, the third target corresponding to a third participant, responsive to determining that the sound source location corresponds to the first target; and

the instructions to select the first sub-frame comprise instructions to select the first sub-frame to include a depiction of the third target.

10. The non-transitory computer readable medium of claim 9 , wherein the instructions to select the second sub-frame comprise instructions to select the second sub-frame to exclude the depiction of the third target.

11. The non-transitory computer readable medium of claim 6 , wherein the instructions further comprise a neural network in which the facial weight value and the first talker weight value each apply to one or more nodes.

12. A teleconferencing endpoint, comprising:

a network interface;

a camera device;

a microphone device;

a processor, the processor coupled to the network interface, the camera device and the microphone device;

a memory, the memory storing instructions executable by the processor, wherein the instructions comprise instructions to:

receive, from the camera device, a video data frame;

detect a presence, within the video data frame, of data indicative of a face;

designate a portion of the video data frame as a target region based on the data indicative of the face;

receive, from the microphone device, an audio data frame, the audio data frame associated in time with the video data frame, the audio data frame indicative of a sound source location;

update an audio-visual map, the audio-visual map corresponding to a plurality of earlier video data frames and audio data frames associated therewith, wherein the instructions to update the audio-visual map comprise instructions to:

determine that the target region corresponds to a first target of the audio-visual map, the first target corresponding to a meeting participant,

increment a facial weight value of the first target, responsive to determining that the target region corresponds to the first target,

determine that the sound source location corresponds to the first target, and

increment a first talker weight value of the first target, responsive to determining that the sound source location corresponds to the first target;

compare the first talker weight value of the first target to a second talker weight value of a second target of the audio-visual map, the second target corresponding to a second participant;

increment a first conversational weight value of the first target and a second conversational weight value of the second target when the first talker weight value of the first target and second talker weight value of the second target each exceed a first threshold and an absolute value of a difference between the first talker weight value of the first target and the second talker weight value of the second target does not exceed a second threshold;

select one or more sub-frames of the video data frame, wherein the instructions to select one or more subframes comprise instructions to:

select a first sub-frame depicting the first target when the facial weight value exceeds a third threshold, and

select a second sub-frame depicting the first target when the first talker weight value exceeds a fourth threshold,

select a third sub-frame depicting the first target and the second target when the first conversational weight value of the first target and the second conversational weight value of the second target each exceed a fifth threshold; and

transmit an audio-video stream containing one or more selected sub-frames to a remote endpoint using the network interface.

13. The teleconferencing endpoint of claim 12 , wherein the first sub-frame and the second sub-frame are different.

14. The teleconferencing endpoint of claim 12 , wherein the first sub-frame and the second sub-frame partially intersect.

15. The teleconferencing endpoint of claim 12 , wherein:

the instructions to update the audio-visual map further comprise instructions to decrement a third talker weight value of a third target of the audio-visual map, the third target corresponding to a third participant, responsive to determining that the sound source location corresponds to the first target; and

the instructions to select the first sub-frame comprise instructions to select the first sub-frame to include a depiction of the third target.

16. The teleconferencing endpoint of claim 15 , wherein the instructions to select the second sub-frame comprise instructions to select the second sub-frame to exclude the depiction of the third target.

17. The teleconferencing endpoint of claim 12 , wherein the instructions further comprise a neural network in which the facial weight value and the first talker weight value each apply to one or more nodes.