Multi-device, multi-channel attention for speech and audio analytics applications
Some disclosed methods involve receiving sensor data, including microphone data from each of a plurality of devices in the environment, producing an input embedding vector corresponding to each sensor, producing a device-wise context vector corresponding to each device, obtaining ground truth data and comparing each device-wise context vector with the ground truth data, to produce a comparison result. The comparing may involve an attention-based process. Some disclosed methods involve generating one or more current output analytics tokens based, at least in part, on the comparison result and controlling the operation of at least one device based, at least in part, in the one or more current output analytics tokens. The controlling may involve controlling at least one of a speaker operation or a microphone operation.
1 . A method, comprising:
receiving, by a control system, sensor data from each of a plurality of sensors in an environment, the plurality of sensors corresponding to a plurality of devices in the environment, the sensor data including microphone data;
for each device:
producing, by the control system, input embedding vectors corresponding to the plurality of sensors of the device;
producing, by the control system, a device-wise context vector from the input embedding vectors for the sensors of the device, resulting in a plurality of device-wise context vectors, wherein producing the plurality of device-wise context vectors involves integrating the input embedding vectors to produce a plurality of cross-channel context vectors, wherein a cross-channel context vector of a first channel is based, at least in part, on channel self-context vectors of at least a second channel and a third channel;
inputting, by the control system, the device-wise context vectors into a machine learning model, wherein the machine learning model includes an attention mechanism;
predicting, by the machine learning model, one or more output analytics;
controlling, by the control system, the operation of at least one device of the plurality of devices in the environment based, at least in part, on the one or more output analytics, wherein the controlling involves controlling at least one of a loudspeaker operation or a microphone operation.
2 . The method of claim 1 , wherein the controlling involves controlling one or more of an automatic speech recognition (ASR) process, an acoustic scene analysis (ASA) process, a talker identification process or a Sound Event Classification (SEC) process.
3 . The method of claim 1 , further comprising:
obtaining, by the control system, one or more prior analytics output tokens within the length of a context window; and
generating, by the control system, an output embedding vector corresponding to the one or more prior analytics output tokens, wherein the ground truth data comprises the one or more prior analytics output tokens.
4 . The method of claim 3 , wherein the one or more output analytics tokens comprise one or more prior analytics output tokens corresponding to an active noise cancellation process.
5 . The method of claim 1 , wherein one or more aspects of the method is implemented via a trained neural network.
6 . The method of claim 5 , wherein the trained neural network comprises a trained attention-based neural network.
7 . The method of claim 1 , wherein the control system is configured to implement a multi-channel neural context encoder for integrating the plurality of input embedding vectors.
8 . The method of claim 7 , wherein the multi-channel neural context encoder comprises a trained attention-based neural network.
9 . The method of claim 1 , further comprising producing a first channel-wise context vector based, at least in part, on a cross-channel context vector and a channel self-context vector.
10 . The method of claim 9 , wherein producing the first channel-wise context vector involves using the channel self-context vector as a query and the cross-channel context vector as key and value inputs.
11 . The method of claim 1 , wherein producing the device-wise context vector involves pooling the plurality of channel-wise context vectors.
12 . One of more non-transitory media having software stored thereon, the software including instructions for controlling one or more devices to implement the method of claim 1 .
13 . An apparatus comprising:
a receiver for receiving sensor data from each of a plurality of sensors in an environment, the plurality of sensors corresponding to a plurality of devices in the environment, the sensor data including microphone data;
a control system for:
producing input embedding vectors corresponding to the plurality of sensors;
producing device-wise context vectors corresponding to the plurality of devices in the environment, the device-wise context vectors produced from the input embeddings for sensors of the device, to produce a plurality of device-wise context vectors, wherein producing the plurality of device-wise context vectors involves integrating the input embedding vectors to produce a plurality of cross-channel context vectors, wherein a cross-channel context vector of a first channel is based, at least in part, on channel self-context vectors of at least a second channel and a third channel;
inputting the device-wise context vectors into a machine learning model, wherein the machine learning model includes an attention mechanism;
controlling the operation of at least one device of the plurality of devices in the environment based, at least in part, on one or more output analytics predicted by the machine learning model, wherein the controlling involves controlling at least one of a loudspeaker operation or a microphone operation.