Distributed teleconferencing using personalized enhancement models
This document relates to distributed teleconferencing. Some implementations can employ personalized enhancement models to enhance microphone signals for participants in a call. Further implementations can perform proximity-based mixing, where microphone signals received from devices in a particular room can be omitted from playback signals transmitted to other devices in the same room. These techniques can allow enhanced call quality for teleconferencing sessions where co-located users can employ their own devices to participate in a call with other users.
1. A method comprising:
receiving a first microphone signal from a first device during a communication session, the first microphone signal being enhanced by a first personalized enhancement model that uses self-attention processing to attenuate components of the first microphone signal including first signals other than the first microphone signal during the communication session, wherein the first personalized enhancement model comprises:
a first encoder configured to process first input features using first multi-head attention to generate first intermediate representations;
a first decoder configured to generate first masking parameters from the first intermediate representations where the first masking parameters are used to selectively enhance first speech from a first user while attenuating other first audio components;
receiving a second microphone signal from a second device that is co-located with the first device during the communication session, the second microphone signal being enhanced by a second personalized enhancement model that uses self-attention processing to attenuate components of the second microphone signal including second signals other than the second microphone signal during the communication session, wherein the second personalized enhancement model comprises:
a second encoder configured to process second input features using second multi-head attention to generate second intermediate representations;
a second decoder configured to generate second masking parameters from the second intermediate representations where the second masking parameters are used to selectively enhance second speech from a second user while attenuating other second audio components;
mixing the first microphone signal with the second microphone signal to obtain a playback signal;
sending the playback signal to a third device that is participating in a call with the first device and the second device; and
sending another playback signal to at least one of the first device or the second device, the another playback signal omitting the first microphone signal and the second microphone signal.
2. The method of claim 1 , further comprising:
detecting that the first device is co-located with the second device; and
omitting the first microphone signal and the second microphone signal from the another playback signal responsive to determining that the first device is co-located with the second device.
3. The method of claim 2 , wherein the detecting is performed using ultrasound communication between the first device and the second device to determine that both devices are in a same room.
4. The method of claim 1 , further comprising:
synchronizing a first speaker of the first device with a second speaker of the second device; and
playing the playback signal using the synchronized first speaker and second speaker.
5. A computing device comprising:
a processor; and
a storage medium storing instructions which, when executed by the processor, cause the computing device to:
receive a first microphone signal from a first device during a communication session, the first microphone signal being enhanced by a first personalized enhancement model that uses self-attention processing to attenuate components of the first microphone signal including first signals other than the first microphone signal during the communication session, wherein the first personalized enhancement model comprises:
a first encoder configured to process first input features using first multi-head attention to generate first intermediate representations;
a first decoder configured to generate first masking parameters from the first intermediate representations where the first masking parameters are used to selectively enhance first speech from a first user while attenuating other first audio components;
receive a second microphone signal from a second device that is co-located with the first device during the communication session, the second microphone signal being enhanced by a second personalized enhancement model that uses self-attention processing to attenuate components of the second microphone signal including second signals other than the second microphone signal during the communication session, wherein the second personalized enhancement model comprises:
a second encoder configured to process second input features using second multi-head attention to generate second intermediate representations;
a second decoder configured to generate second masking parameters from the second intermediate representations where the second masking parameters are used to selectively enhance second speech from a second user while attenuating other second audio components;
mix the first microphone signal with the second microphone signal to obtain a playback signal;
send the playback signal to a third device that is participating in a call with the first device and the second device; and
send another playback signal to at least one of the first device or the second device, the another playback signal omitting the first microphone signal and the second microphone signal.
6. The computing device of claim 5 , wherein the instructions, when executed by the processor, cause the computing device to:
detect that the first device is co-located with the second device; and
omit the first microphone signal and the second microphone signal from the another playback signal responsive to determining that the first device is co-located with the second device.
7. The computing device of claim 6 , wherein the detecting is performed using ultrasound communication between the first device and the second device to determine that both devices are in a same room.
8. The computing device of claim 5 , wherein the instructions, when executed by the processor, cause the computing device to:
synchronize a first speaker of the first device with a second speaker of the second device; and
play the playback signal using the synchronized first speaker and second speaker.
9. A system comprising:
means for receiving a first microphone signal from a first device during a communication session, the first microphone signal being enhanced by a first personalized enhancement model that uses self-attention processing to attenuate components of the first microphone signal including first signals other than the first microphone signal during the communication session, wherein the first personalized enhancement model comprises:
a first encoder configured to process first input features using first multi-head attention to generate first intermediate representations;
a first decoder configured to generate first masking parameters from the first intermediate representations where the first masking parameters are used to selectively enhance first speech from a first user while attenuating other first audio components;
means for receiving a second microphone signal from a second device that is co-located with the first device during the communication session, the second microphone signal being enhanced by a second personalized enhancement model that uses self-attention processing to attenuate components of the second microphone signal including second signals other than the second microphone signal during the communication session, wherein the second personalized enhancement model comprises:
a second encoder configured to process second input features using second multi-head attention to generate second intermediate representations;
a second decoder configured to generate second masking parameters from the second intermediate representations where the second masking parameters are used to selectively enhance second speech from a second user while attenuating other second audio components;
means for mixing the first microphone signal with the second microphone signal to obtain a playback signal;
means for sending the playback signal to a third device that is participating in a call with the first device and the second device; and
means for sending another playback signal to at least one of the first device or the second device, the another playback signal omitting the first microphone signal and the second microphone signal.
10. The system of claim 9 , the system further comprising:
means for detecting that the first device is co-located with the second device; and
means for omitting the first microphone signal and the second microphone signal from the another playback signal responsive to determining that the first device is co-located with the second device.
11. The system of claim 10 , wherein the detecting is performed using ultrasound communication between the first device and the second device to determine that both devices are in a same room.
12. The system of claim 9 , the system further comprising:
means for synchronizing a first speaker of the first device with a second speaker of the second device; and
means for playing the playback signal using the synchronized first speaker and second speaker.