IP Library Granted Patent US 10,679,621
Granted Patent B1
US 10,679,621 · App. 15/927,764 · Granted Jun 9, 2020

Speech processing optimizations based on microphone array

Inventors: Shiva Kumar Sundaram (Fremont, CA); Minhua Wu (San Jose, CA); Anirudh Raju (San Jose, CA); Spyridon Matsoukas (Hopkinton, MA); Arindam Mandal (Redwood City, CA); Kenichi Kumatani (San Jose, CA)
Assignee: Amazon Technologies, Inc.
G10L15/22G06F40/40G10L15/187G10L15/26G10L15/30G10L21/0208H04R3/005G10L2015/088G10L2015/223G10L2021/02166H04W4/025
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,679,621
App. No.
15/927,764
Granted
Jun 9, 2020
Kind
B1
Abstract

Systems and methods for utilizing microphone array information for acoustic modeling are disclosed. Audio data may be received from a device having a microphone array configuration. Microphone configuration data may also be received that indicates the configuration of the microphone array. The microphone configuration data may be utilized as an input vector to an acoustic model, along with the audio data, to generate phoneme data. Additionally, the microphone configuration data may be utilized to train and/or generate acoustic models, select an acoustic model to perform speech recognition with, and/or to improve trigger sound detection.

Claims (83)

1. A system comprising:

one or more processors; and

computer-readable media storing computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:

receiving, from a device having a microphone array, audio data representing a user utterance;

receiving microphone configuration data indicating a configuration of the microphone array, the configuration including at least one of a number of microphones in the microphone array, a spacing of the microphones, a geometry of the microphones, or a type of the microphones;

selecting an acoustic model from multiple acoustic models based at least in part on the configuration;

generating an input vector from the microphone configuration data, the input vector configured to be input into the acoustic model;

identifying a portion of the audio data associated with a time interval;

inputting, into the acoustic model:

a feature vector associated with the portion of the audio data; and

the input vector; and

generating, using the acoustic model, the feature vector, and the input vector, phoneme data indicating a probability that a phoneme corresponds to the portion of the audio data.

2. The system of claim 1 , the operations further comprising:

generating a trigger-sound component configured to transition at least a portion of the device from a first state to a second state when the device determines that a trigger sound has been uttered, wherein the trigger-sound component is configured to detect the trigger sound using audio data generated by the microphones; and

sending, to the device, configuration data representing the trigger-sound component for use by the device to detect the trigger sound.

3. The system of claim 1 , the operations further comprising:

identifying a first parameter of the acoustic model;

determining, from the microphone configuration data, that the configuration of the microphone array differs from the microphone array configuration associated with the first parameter;

generating a second parameter based on the microphone configuration data; and

causing the first parameter to be replaced by the second parameter.

4. A method, comprising:

receiving audio data generated by a device having at least one microphone;

receiving microphone configuration data indicating a physical arrangement of the at least one microphone;

selecting an acoustic model from multiple acoustic models based at least in part on the physical arrangement;

inputting, into the acoustic model, a value corresponding to a portion of the audio data and an input vector representing the microphone configuration data; and

generating, using the acoustic model and based at least in part on the value and the input vector, output data representing the portion of the audio data.

5. The method of claim 4 , further comprising generating, utilizing the acoustic model and based at least in part on the output data, text data representing the audio data.

6. The method of claim 4 , wherein the microphone configuration data comprises at least one of a number of the at least one microphone, a spacing of the at least one microphone from another component of the device, or a type of the at least one microphone.

7. The method of claim 4 , further comprising:

identifying a first parameter of the acoustic model that is associated with microphone arrangement;

determining, from the microphone configuration data, that the physical arrangement of the at least one microphone differs from the microphone arrangement associated with the first parameter;

generating a second parameter based at least in part on the microphone configuration data; and

causing the first parameter to be replaced by the second parameter.

8. The method of claim 4 , wherein the input vector comprises a first input vector, and further comprising:

receiving, via the device, global-positioning data indicating a location of the device;

inputting a second input vector representing the global-positioning data into the acoustic model; and

wherein generating the output data is based at least in part on the second input vector.

9. The method of claim 4 , wherein the input vector comprises a first input vector, and further comprising:

receiving, via the device, device-location data indicating a location of the device within an environment;

inputting a second input vector representing the device location data into the acoustic model; and

wherein generating the output data is based at least in part on the second input vector.

10. The method of claim 4 , wherein the input vector comprises a first input vector, and further comprising:

receiving, via the device, speaker-location data indicating a location, relative to the device, of a user that provided the user utterance;

inputting a second input vector representing the speaker-location data into the acoustic model; and

wherein generating the output data is based at least in part on the second input vector.

11. The system of claim 4 , the operations further comprising:

receiving a device identifier associated with the device;

sending request data for the microphone configuration data to a remote system associated with the device identifier; and

wherein receiving the microphone configuration data comprises receiving the microphone configuration data from the remote system.

12. The method of claim 4 , the operations further comprising:

determining that the physical arrangement differs from physical arrangements associated with the acoustic models;

determining a reference physical arrangement associated with the acoustic model has a first arrangement characteristic that corresponds to a second arrangement characteristic of the physical arrangement; and

wherein selecting the acoustic model comprises selecting the acoustic model based at least in part on the first arrangement characteristic corresponding to the second arrangement characteristics.

13. A system, comprising:

one or more processors; and

computer-readable media storing computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:

receiving, from a device having at least one microphone, audio data;

receiving microphone configuration data indicating a physical arrangement of the at least one microphone;

selecting, based at least in part on the microphone configuration data, an acoustic model of acoustic models; and

associating, utilizing the acoustic model, a sound with the audio data.

14. The system of claim 13 , wherein associating the sound with the audio data comprises associating the sound with the audio data based at least in part on:

inputting a value corresponding to a portion of the audio data associated with a time interval and an input vector representing the microphone configuration data into the acoustic model; and

generating, using the acoustic model and based at least in part on the input vector, output data indicating a probability that the sound corresponds to the portion of the audio data.

15. The system of claim 13 , the operations further comprising generating the microphone configuration data based at least in part on microphone configuration information associated with the device.

16. The system of claim 13 , wherein the microphone configuration data comprises at least one of a number of the at least one microphone, a spacing of the at least one microphone with respect to at least one other component of the device, or a type of the at least one microphone.

17. The system of claim 13 , wherein the acoustic model represents an association between a portion of the audio data and a sound, and the operations further comprising:

identifying a first parameter of the acoustic model that is associated with microphone arrangement;

determining, from the microphone configuration data, that the physical arrangement of the at least one microphone differs from the microphone arrangement associated with the first parameter;

generating a second parameter based at least in part on the microphone configuration data; and

causing the first parameter to be replaced by the second parameter.

18. The system of claim 13 , wherein the at least one microphone comprises a first microphone and a second microphone, the audio data comprises a first audio signal from the first microphone and a second audio signal from the second microphone, and the operations further comprising:

processing, based at least in part on the microphone configuration data, the first audio signal and the second audio signal to generate optimized audio data; and

wherein associating the sound with the audio data comprises associating the sound with the audio data based at least in part on the optimized audio data.

19. The system of claim 13 , the operations further comprising:

generating a trigger-sound component configured to transition at least a portion of the device from a first state to a second state when the device determines that a trigger sound has been uttered, wherein the trigger-sound component is configured to detect the trigger sound using audio data generated by the at least one microphone configured in the physical arrangement; and

sending, to the device, configuration data representing the trigger-sound component for use by the device to detect the trigger sound.

20. The system of claim 13 , the operations further comprising:

receiving, via the device, contextual data including at least one of:

global positioning data indicating a geographic position of the device;

device-location data indicating a first location of the device within an environment; or

speaker-location data indicating a second location of a user that provided the user utterance;

inputting an input vector representing the contextual data into the acoustic model; and

generating, via the acoustic model, output data based at least in part on the input vector and the audio data.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 21, 2018
From: SUNDARAM, SHIVA KUMAR; WU, MINHUA; RAJU, ANIRUDH; MATSOUKAS, SPYRIDON; MANDAL, ARINDAM; KUMATANI, KENICHI
To: AMAZON TECHNOLOGIES, INC.
Reel/Frame 045306/0298 →
Cited By (5)
US 12,375,841 US 12,393,717 US 12,451,125 US 12,603,088 US 12,707,220