Audio selection based on user engagement
In one embodiment, a method includes receiving audio input data from a microphone array of at least two microphones. The audio input data is generated by a first sound source at a first location and a second sound source at a second location. The method also includes calculating a first engagement metric for the first sound source and a second engagement metric for the second sound source. The first engagement metric approximates an interest level of a receiving user for the first sound source, and the second engagement metric approximates an interest level from the receiving user for the second sound source. The method also includes determining that the first engagement metric is greater than the second engagement metric, and processing the audio input data to generate an audio output signal. The audio output signal may amplify sound generated by the first sound source relative to the second sound source.
1. A method comprising:
receiving, during an audio-video communication session, audio input data from a microphone array comprising at least two microphones, wherein the audio input data is generated by a first sound source at a first location within an environment and a second sound source at a second location within the environment;
determining a first classification for the first sound source and a second classification for the second sound source;
predicting a first engagement metric for the first sound source and a second engagement metric for the second sound source, wherein:
the first engagement metric is based on the first classification and the second engagement metric is based on the second classification;
the first engagement metric approximates an interest level of a receiving user for the first sound source; and
the second engagement metric approximates an interest level from the receiving user for the second sound source;
determining that the first engagement metric is greater than the second engagement metric;
processing the audio input data to generate an audio output signal, wherein the audio output signal amplifies sound generated by the first sound source and attenuates sound generated by the second sound source; and
sending the audio output signal to a computing device associated with the receiving user.
2. The method of claim 1 , wherein the first classification for the first sound source is a human voice, and wherein the second classification for the second sound source is a non-human sound.
3. The method of claim 1 , wherein the determining the first classification and the second classification is based on information received from a descriptive model for the audio-video communication session that comprises one or more descriptive characteristics about (1) an environment associated with the audio-video communication session; (2) one or more people within the environment, or (3) one or more contextual elements associated with the audio-video communication session.
4. The method of claim 1 , wherein the processing the audio input data comprises acoustically beamforming a first audio input signal generated by the first source and a second audio input signal generated by the second source, wherein the acoustical beamforming comprises time delaying the second audio input signal such that the first sound source is amplified and the second sound source is attenuated.
5. The method of claim 1 , wherein the first engagement metric and the second engagement metric are calculated based a descriptive model for the audio-video communication session that comprises one or more descriptive characteristics about (1) an environment associated with the audio-video communication session; (2) one or more people within the environment, or (3) one or more contextual elements associated with the audio-video communication session.
6. The method of claim 1 , further comprising:
accessing a social graph comprising a plurality of nodes and a plurality of edges connecting the nodes, wherein:
a first node corresponds to the receiving user;
a second node corresponds to an entity associated with the first sound source; and
an edge between the first node and the second node represents a relationship between the receiving user and the entity; and
increasing the first engagement metric based on the edge between the first node and the second node.
7. The method of claim 1 , wherein the first engagement is calculated at least in part based on a count of words spoken by the first sound source, a distance between the first sound source and the microphone array, or an amount of time the first sound source has been present in the environment during the audio-video communication session; and
the second engagement is calculated at least in part based on a count of words spoken by the second sound source, a distance between the second sound source and the microphone array, or an amount of time the second sound source has been present in the environment during the audio-video communication session.
8. A computer-readable non-transitory storage medium embodying software that is operable when executed to:
receive, during an audio-video communication session, audio input data from a microphone array comprising at least two microphones, wherein the audio input data is generated by a first sound source at a first location within an environment and a second sound source at a second location within the environment;
determine a first classification for the first sound source and a second classification for the second sound source;
predict a first engagement metric for the first sound source and a second engagement metric for the second sound source, wherein:
the first engagement metric is based on the first classification and the second engagement metric is based on the second classification;
the first engagement metric approximates an interest level of a receiving user for the first sound source; and
the second engagement metric approximates an interest level from the receiving user for the second sound source;
determine that the first engagement metric is greater than the second engagement metric;
process the audio input data to generate an audio output signal, wherein the audio output signal amplifies sound generated by the first sound source and attenuates sound generated by the second sound source; and
send the audio output signal to a computing device associated with the receiving user.
9. The computer-readable non-transitory storage medium of claim 8 , wherein the first classification for the first sound source is a human voice, and wherein the second classification for the second sound source is a non-human sound.
10. The media computer-readable non-transitory storage medium of claim 8 , wherein the determining the first classification and the second classification is based on information received from a descriptive model for the audio-video communication session that comprises one or more descriptive characteristics about (1) an environment associated with the audio-video communication session; (2) one or more people within the environment, or (3) one or more contextual elements associated with the audio-video communication session.
11. The computer-readable non-transitory storage medium of claim 8 , wherein the processing the audio input data comprises acoustically beamforming a first audio input signal generated by the first source and a second audio input signal generated by the second source, wherein the acoustical beamforming comprises time delaying the second audio input signal such that the first sound source is amplified and the second sound source is attenuated.
12. The computer-readable non-transitory storage medium of claim 8 , wherein the first engagement metric and the second engagement metric are calculated based a descriptive model for the audio-video communication session that comprises one or more descriptive characteristics about (1) an environment associated with the audio-video communication session; (2) one or more people within the environment, or (3) one or more contextual elements associated with the audio-video communication session.
13. The computer-readable non-transitory storage medium of claim 8 , wherein the software is further operable when executed to:
access a social graph comprising a plurality of nodes and a plurality of edges connecting the nodes, wherein:
a first node corresponds to the receiving user;
a second node corresponds to an entity associated with the first sound source; and
an edge between the first node and the second node represents a relationship between the receiving user and the entity; and
increase the first engagement metric based on the edge between the first node and the second node.
14. The computer-readable non-transitory storage medium of claim 8 , wherein the first engagement is calculated at least in part based on a count of words spoken by the first sound source, a distance between the first sound source and the microphone array, or an amount of time the first sound source has been present in the environment during the audio-video communication session; and
the second engagement is calculated at least in part based on a count of words spoken by the second sound source, a distance between the second sound source and the microphone array, or an amount of time the second sound source has been present in the environment during the audio-video communication session.
15. A system comprising:
one or more processors; and
a computer-readable non-transitory storage medium coupled to one or more of the processors and comprising instructions operable when executed by one or more of the processors to cause the system to:
receive, during an audio-video communication session, audio input data from a microphone array comprising at least two microphones, wherein the audio input data is generated by a first sound source at a first location within an environment and a second sound source at a second location within the environment;
determine a first classification for the first sound source and a second classification for the second sound source;
predict a first engagement metric for the first sound source and a second engagement metric for the second sound source, wherein:
the first engagement metric is based on the first classification and the second engagement metric is based on the second classification;
the first engagement metric approximates an interest level of a receiving user for the first sound source; and
the second engagement metric approximates an interest level from the receiving user for the second sound source;
determine that the first engagement metric is greater than the second engagement metric;
process the audio input data to generate an audio output signal, wherein the audio output signal amplifies sound generated by the first sound source and attenuates sound generated by the second sound source; and
send the audio output signal to a computing device associated with the receiving user.
16. The system of claim 15 , wherein the first classification for the first sound source is a human voice, and wherein the second classification for the second sound source is a non-human sound.
17. The system of claim 15 , wherein the determining the first classification and the second classification is based on information received from a descriptive model for the audio-video communication session that comprises one or more descriptive characteristics about (1) an environment associated with the audio-video communication session; (2) one or more people within the environment, or (3) one or more contextual elements associated with the audio-video communication session.