Spatial audio and avatar control at headset using audio signals
An audio system in a local area providing an audio signal to a headset of a remote user is presented herein. The audio system identifies sounds from a human sound source in the local area, based in part on sounds detected within the local area. The audio system generates an audio signal for presentation to a remote user within a virtual representation of the local area based in part on a location of the remote user within the virtual representation of the local area relative to a virtual representation of the human sound source within the virtual representation of the local area. The audio system provides the audio signal to a headset of the remote user, wherein the headset presents the audio signal as part of the virtual representation of the local area to the remote user.
1 . A method comprising:
receiving audio data captured, by a first computing system, from a human sound source;
predicting a facial expression based at least in part on the audio data;
causing a second computing system, remote from the first computing system, to play audio, based on the audio data, in relation to a virtual representation of the human sound source; and
causing the second computing system to provide, in a virtual environment and in synchronization with the audio that is played, an additional virtual representation of the predicted expression in relation to the virtual representation of the human sound source, the additional virtual representation being provided responsive to an occluded portion of a face of the human sound source being within a threshold angle of a field of view of the second computing system.
2 . The method of claim 1 , wherein the predicting the facial expression comprises predicting a lip pose or movement for the representation of the human sound source.
3 . The method of claim 1 , wherein the predicting the facial expression comprises predicting the facial expression by applying a machine learning algorithm to the audio data.
4 . The method of claim 1 , further comprising:
selectively adjusting the audio data in response to one or more user inputs;
wherein the causing the second computing system to play the audio comprises causing the second computing system to play the audio based on the adjusted audio data.
5 . The method of claim 1 , wherein the audio that the second computing system is caused to play is modified based on a comparison between a location determined for the second computing system and a location determined for the representation of the human sound source.
6 . The method of claim 1 , wherein the captured audio data is received in response to:
generation of multiple captured audio data instances, from sound sources collocated with the human sound source; and
identifying one of the multiple captured audio data instances, as being from the human sound source, based on matching between the multiple captured audio data instances and data for the human sound source.
7 . The method of claim 1 :
wherein the captured audio data is associated with a location of the human sound source determined by performing beam-steering processing on the captured audio data; and
wherein the audio that the second computing system is caused to play is modified based on the location of the human sound source.
8 . The method of claim 1 , wherein the causing the second computing system to provide the predicted facial expression of the human sound source in synchronization with the played audio includes providing, to the second computing system via a network, visual information indicating the predicted facial expression with synchronization information for synchronizing the predicted facial expression with playing the audio.
9 . The method of claim 1 , wherein the method is performed by the first computing system.
10 . The method of claim 1 , wherein the method is performed by the second computing system.
11 . The method of claim 1 , wherein the method is performed by an intermediary system facilitating communication between the first computing system and the second computing system.
12 . A computer-readable storage medium storing instructions that, when executed by a computing system, cause the computing system to perform a process comprising:
receiving audio data captured, by a first computing system, from a human sound source;
predicting a facial expression;
causing a second computing system, remote from the first computing system, to play audio, based on the audio data, in relation to a virtual representation of the human sound source; and
causing the second computing system to provide, in a virtual environment and in synchronization with the audio that is played, an additional virtual representation of the predicted expression in relation to the virtual representation of the human sound source, the additional virtual representation being provided responsive to an occluded portion of a face of the human sound source being within a threshold angle of a field of view of the second computing system.
13 . The computer-readable storage medium of claim 12 , wherein the predicting the facial expression comprises predicting the facial expression by applying a machine learning algorithm to the audio data.
14 . The computer-readable storage medium of claim 12 , wherein the predicting the facial expression comprises predicting a lip pose or movement for the representation of the human sound source.
15 . The computer-readable storage medium of claim 12 , wherein the audio that the second computing system is caused to play is modified based on a comparison between a location determined for the second computing system and a location determined for the representation of the human sound source.
16 . The computer-readable storage medium of claim 12 , wherein the process is performed by the second computing system.
17 . A computing system comprising:
one or more processors; and
one or more memories storing instructions that, when executed by the one or more processors, cause the computing system to perform a process comprising:
receiving audio data captured, by a first computing system;
predicting a facial expression;
causing a second computing system, remote from the first computing system, to play audio, based on the audio data, in relation to a virtual representation of a human sound source; and
causing the second computing system to provide, in a virtual environment and in synchronization with the audio that is played, an additional virtual representation of the predicted expression in relation to the virtual representation of the human sound source, the additional virtual representation being provided responsive to an occluded portion of a face of the human sound source being within a threshold angle of a field of view of the second computing system.
18 . The computing system of claim 17 :
wherein the captured audio data is associated with a location of the human sound source determined by performing beam-steering processing on the captured audio data; and
wherein the audio that the second computing system is caused to play is modified based on the location of the human sound source.
19 . The computing system of claim 17 , wherein the process is performed by the first computing system.
20 . The computing system of claim 17 , wherein the causing the second computing system to provide the predicted facial expression of the human sound source in conjunction with the played audio includes providing, to the second computing system, visual information indicating the predicted facial expression with synchronization information for synchronizing the predicted facial expression with playing the audio.