Granted Patent B1

US 12,190,875 · App. 17/490,572 · Granted Jan 7, 2025

Inventors: Eli Joshua Fidler (Toronto, CA); Aaron Challenner (Cambridge, MA); Zoe Adams (Orange County, CA); Sree Hari Krishnan Parthasarathi (Toronto, CA); Gengshen Fu (Sharon, MA)

Assignee: Amazon Technologies, Inc.

G10L15/22G10L15/02G10L15/08G10L15/187

Abstract

Claims (105)

1. A device, comprising:

one or more processors; and

non-transitory computer-readable media storing computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:

receiving first audio data representing a first user utterance captured by a microphone of the device;

determining, from a comparison of a first audio signature of the first audio data to a stored audio signature corresponding to a first part of a wakeword, that a portion of the first audio data corresponds to the first part of the wakeword, the wakeword, when detected, causing the device to send the first audio data to a speech processing system, the first part of the wakeword being one or more starting syllables of the wakeword;

performing, in response to the portion of the first audio data corresponding to the first part of the wakeword, automatic speech recognition (ASR) on the first audio data such that first ASR data is generated; and

determining that an entirety of the wakeword is detected in the first audio data.

2. The device of claim 1 , the operations further comprising:

storing first data indicating a first confidence threshold for detecting the first part of the wakeword;

receiving second data indicating when both the first part of the wakeword is detected and the entirety of the wakeword is detected in sample audio data; and

generating, based at least in part on the second data, third data indicating a second confidence threshold for detecting the first part of the wakeword, the second confidence threshold differing from the first confidence threshold.

3. The device of claim 1 , the operations further comprising:

receiving second audio data representing a second user utterance;

detecting the first part of the wakeword in the second audio data;

initiating speech processing on the second audio data in response to detecting the first part of the wakeword in the second audio data such that second data representing a first part of the second user utterance is generated;

determining that the entirety of the wakeword is absent from the second data; and

causing the speech processing to cease in response to the entirety of the wakeword being absent from the second data.

4. The device of claim 1 , the operations further comprising:

in response to determining that the portion of the first audio data corresponds to the first part of the wakeword, selecting a first speech processing component associated with the wakeword to perform speech processing on the first audio data, wherein performing the speech processing is performed by the first speech processing component;

receiving second audio data representing a second user utterance;

detecting a first portion of a keyword other than the wakeword from the second audio data;

selecting a second speech processing component associated with the keyword in response to detecting the first portion of the keyword; and

performing, using the second speech processing component, speech processing on the second audio data.

5. A method, comprising:

receiving first audio data representing a first user utterance;

determining, based at least in part on a first audio signature of the first audio data corresponding to a stored audio signature of a first part of a first keyword, that at least a portion of the first audio data corresponds to the first part of the first keyword;

initiating generation of first data indicating words included in the first user utterance;

detecting an entirety of the first keyword from the first audio data; and

causing generation of the first data to proceed based at least in part on the entirety of the first keyword being detected.

6. The method of claim 5 , further comprising:

receiving second audio data representing a second user utterance;

detecting, at a wakeword engine of a device that received the first audio data, the first part of the first keyword in the second audio data;

initiating, on the device, speech processing on the second audio data based at least in part on detecting the first part of the first keyword in the second audio data;

determining, at the wakeword engine, that the entirety of the first keyword is undetected; and

causing the speech processing to cease based at least in part on the entirety of the first keyword being undetected.

7. The method of claim 5 , further comprising:

receiving second audio data representing a second user utterance;

detecting the first part of the first keyword in the second audio data;

performing speech processing on the second audio data based at least in part on detecting the first part of the first keyword in the second audio data such that second data representing the second user utterance is generated;

determining, from the second data, that the entirety of the first keyword is absent from the second data; and

causing the speech processing to cease based at least in part on the entirety of the first keyword being absent from the second data.

8. The method of claim 5 , further comprising:

based at least in part on determining that the portion of the first audio data corresponds to the first part of the first keyword, selecting a first speech processing component associated with the first keyword to perform speech processing on the first audio data, wherein performing the speech processing is performed by the first speech processing component;

receiving second audio data representing a second user utterance;

detecting a first portion of a second keyword from the second audio data;

selecting a second speech processing component associated with the second keyword based at least in part on detecting the first portion of the second keyword; and

performing, using the second speech processing component, speech processing on the second audio data.

9. The method of claim 5 , further comprising:

storing second data indicating a first confidence threshold for detecting the first part of the first keyword;

receiving third data indicating when both the first part of the first keyword is detected and the entirety of the first keyword is detected in sample audio data; and

generating, based at least in part on the third data, fourth data indicating a second confidence threshold for detecting the first part of the first keyword, the second confidence threshold differing from the first confidence threshold.

10. The method of claim 5 , wherein:

determining that the at least the portion of the first audio data corresponds to the first part of the first keyword is performed utilizing a wakeword model of a wakeword engine disposed on a device that received the first audio data; and

detecting the entirety of the first keyword is performed by the wakeword engine based at least in part on the wakeword model determining that the at least the portion of the first audio data corresponds to the first keyword.

11. The method of claim 5 , wherein:

determining that the at least the portion of the first audio data corresponds to the first part of the first keyword is performed utilizing a first wakeword model of a wakeword engine disposed on a device that received the first audio data and configured to detect the first part of the first keyword; and

detecting the entirety of the first keyword is performed by the wakeword engine utilizing a second wakeword model configured to detect the entirety of the first keyword.

12. The method of claim 5 , further comprising:

receiving second audio data representing a second user utterance;

detecting, prior to sending the second audio data to a wakeword engine of a device that received the first audio data, a feature of the second audio data that corresponds to the first part of the first keyword;

initiating, on the device, speech processing on the second audio data based at least in part on detecting the feature;

detecting, at the wakeword engine, the entirety of the wakeword; and

causing the speech processing to continue based at least in part on the entirety of the first keyword being detected at the wakeword engine.

13. A device, comprising:

one or more processors; and

non-transitory computer-readable media storing computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:

receiving first audio data representing a first user utterance;

determining, based at least in part on a first audio signature of the first audio data corresponding to a stored audio signature of a first part of a first keyword, that at least a portion of the first audio data corresponds to the first part of the first keyword;

initiating generation of first data indicating words included in the first user utterance;

detecting an entirety of the first keyword from the first audio data; and

causing generation of the first data to proceed based at least in part on the entirety of the first keyword being detected.

14. The device of claim 13 , the operations further comprising:

receiving second audio data representing a second user utterance;

detecting, at a wakeword engine of a device that received the first audio data, the first part of the first keyword in the second audio data;

initiating, on the device, speech processing on the second audio data based at least in part on detecting the first part of the first keyword in the second audio data;

determining, at the wakeword engine, that the entirety of the first keyword is undetected; and

causing the speech processing to cease based at least in part on the entirety of the first keyword being undetected.

15. The device of claim 13 , the operations further comprising:

receiving second audio data representing a second user utterance;

detecting the first part of the first keyword in the second audio data;

performing speech processing on the second audio data based at least in part on detecting the first part of the first keyword in the second audio data such that second data representing the second user utterance is generated;

determining, from the second data, that the entirety of the first keyword is absent from the second data; and

causing the speech processing to cease based at least in part on the entirety of the first keyword being absent from the second data.

16. The device of claim 13 , the operations further comprising:

based at least in part on determining that the portion of the first audio data corresponds to the first part of the first keyword, selecting a first speech processing component associated with the first keyword to perform speech processing on the first audio data, wherein performing the speech processing is performed by the first speech processing component;

receiving second audio data representing a second user utterance;

detecting a first portion of a second keyword from the second audio data;

selecting a second speech processing component associated with the second keyword based at least in part on detecting the first portion of the second keyword; and

performing, using the second speech processing component, speech processing on the second audio data.

17. The device of claim 13 , the operations further comprising:

storing second data indicating a first confidence threshold for detecting the first part of the first keyword;

receiving third data indicating when both the first part of the first keyword is detected and the entirety of the first keyword is detected in sample audio data; and

generating, based at least in part on the third data, fourth data indicating a second confidence threshold for detecting the first part of the first keyword, the second confidence threshold differing from the first confidence threshold.

18. The device of claim 13 , wherein:

determining that the at least the portion of the first audio data corresponds to the first part of the first keyword is performed utilizing a wakeword model of a wakeword engine disposed on a device that received the first audio data; and

detecting the entirety of the first keyword is performed by the wakeword engine based at least in part on the wakeword model determining that the at least the portion of the first audio data corresponds to the first keyword.

19. The device of claim 13 , wherein:

determining that the at least the portion of the first audio data corresponds to the first part of the first keyword is performed utilizing a first wakeword model of a wakeword engine disposed on a device that received the first audio data and configured to detect the first part of the first keyword; and

detecting the entirety of the first keyword is performed by the wakeword engine utilizing a second wakeword model configured to detect the entirety of the first keyword.

20. The device of claim 13 , the operations further comprising:

receiving second audio data representing a second user utterance;

detecting, prior to sending the second audio data to a wakeword engine of a device that received the first audio data, a feature of the second audio data that corresponds to the first part of the first keyword;

initiating, on the device, speech processing on the second audio data based at least in part on detecting the feature;

detecting, at the wakeword engine, the entirety of the wakeword; and

causing the speech processing to continue based at least in part on the entirety of the first keyword being detected at the wakeword engine.

Assignments (1)

References Cited (15)

Cited By (2)

US 12,412,578 US 12,548,557