IP Library › Granted Patent US 11,830,485
Granted Patent B2
US 11,830,485 · App. 16/216,438 · Granted Nov 28, 2023

Multiple speech processing system with synthesized speech styles

Inventors: Abhay Gupta (Los Altos, CA); Timothy Whalin (Seattle, WA)
Assignee: Amazon Technologies, Inc.
G10L15/22B60K35/00G06F3/167G10L15/08G10L15/32B60K2370/148G10L2015/088G10L2015/223G10L2015/228
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,830,485
App. No.
16/216,438
Granted
Nov 28, 2023
Kind
B2
Abstract

A device having a voice-based interface activates or “wakes” when it detects an utterance that includes a wakeword; the device may be installed in a vehicle, such as an automobile. The device may distinguish between different wakewords; a different speech processing system may be associated with each wakeword, and each speech processing engine may have its own speech style and associated applications and functions.

Claims (91)

1. A method for processing a spoken command, the method comprising:

receiving, from a device, first audio data representing an utterance;

determining that the first audio data includes:

a first representation of a first wakeword corresponding to a first speech processing system, and

a second representation of a command;

determining that the first wakeword corresponds to a first speech processing system, the first speech processing system including a first natural-language understanding (NLU) component and a first text-to-speech (TTS) component corresponding to a first speech style indicating output generated by the first speech processing system;

processing, using the first NLU component, the first audio data to determine intent data corresponding to the command;

causing the device to output light in a first color when processing the first audio data using the first NLU component, wherein processing by a second speech processing system is associated with output of a second color different from the first color;

determining a first score corresponding to a first ability of the first speech processing system to respond to the intent data;

determining a second score corresponding to a second ability of the second speech processing system to respond to the intent data, the second speech processing system including a second TTS component corresponding to a second speech style different from the first speech style and indicating output generated by the second speech processing system;

based on the first score and the second score, sending the intent data to the second speech processing system;

receiving, from the second speech processing system in response to the intent data, first text data responsive to the intent data and an indication of an action to be performed;

generating, using the first text data and the first TTS component, second audio data representing synthesized speech of the first text data and corresponding to the first speech style;

sending the second audio data to the device for output; and

causing the device to perform the action.

2. The method of claim 1 , further comprising:

after determining that the first audio data includes the first representation of the first wakeword, sending a first instruction to output an indication of the first speech processing system; and

prior to sending the second audio data, sending a second instruction to output audio representing a name of the second speech processing system.

3. The method of claim 1 , further comprising:

receiving third audio data representing second speech;

determining that the third audio data includes:

a second representation of a second wakeword, the second wakeword corresponding to the second speech processing system; and

a second representation of a second command;

sending, to the second speech processing system, the third audio data; and

receiving, from the second speech processing system, second text data responsive to the second command.

4. A computer-implemented method comprising:

receiving, from a device, first audio data representing an utterance;

determining, based on a first wakeword detected in the first audio data, to process the first audio data using a first speech processing system, the first speech processing system including a first natural-language understanding (NLU) component and a first text-to-speech (TTS) component corresponding to a first speech style indicating output generated by the first speech processing system;

processing, using the first NLU component, the first audio data to determine intent data corresponding to an intent associated with the utterance;

causing the device to output light in a first color when processing the first audio data using the first NLU component, wherein processing by a second speech processing system is associated with output of a second color different from the first color;

determining a first ability of the first speech processing system to respond to the intent;

determining a second ability of the second speech processing system to respond to the intent, the second speech processing system including a second TTS component corresponding to a second speech style different from the first speech style and indicating output generated by the second speech processing system;

based at least in part on the second ability and the first ability, causing the second speech processing system to process the intent data;

receiving, from the second speech processing system in response to processing the intent data, first response data;

generating, using the first response data and the first TTS component, second audio data corresponding to the first speech style; and

causing the device to output the second audio data.

5. The computer-implemented method of claim 4 , further comprising:

in response to determining to process the first audio data using the first speech processing system, causing the device to output a first indication associated with the first speech processing system, wherein the second speech processing system is associated with a second indication different from the first indication.

6. The computer-implemented method of claim 4 , further comprising:

determining that the intent corresponds to a prior command;

determining that a user profile includes second response data corresponding to the prior command and corresponding to the first speech processing system; and

sending, to the second speech processing system, the second response data.

7. The computer-implemented method of claim 4 , further comprising:

sending, to the first speech processing system, the intent data;

receiving, from the first speech processing system, second response data; and

outputting, based on the second response data and using the first speech processing system, third audio data.

8. The computer-implemented method of claim 4 , further comprising:

receiving, from the second speech processing system, an indication of an action to be performed by the device, wherein the device is associated with a vehicle; and

causing the device to perform the action, wherein performing the action includes causing a change in a state of the vehicle.

9. The computer-implemented method of claim 8 , wherein causing a change in a state of the vehicle includes one or more of raising or lowering a window, setting a thermostat, adjusting a seat, locking or unlocking doors, or turning on or turning off lights.

10. The computer-implemented method of claim 4 , further comprising:

in response to detecting the first wakeword, causing the device to output audio identifying the first speech processing system.

11. The computer-implemented method of claim 4 , further comprising:

determining that the first audio data additionally represents a second intent corresponding to the second speech processing system;

processing the first audio data using a second NLU component corresponding to the second speech processing system, to determine second intent data corresponding to the second intent;

generating, using the second TTS component, second response data; and

causing the device to output third audio data corresponding to the second response data.

12. A computing device comprising:

at least one processor; and

at least one memory including instructions that, when executed by the at least one processor, cause the computing device to:

receive, from a device, first audio data representing an utterance;

determine, based on a first wakeword detected in the first audio data, to process the first audio data using a first speech processing system, the first speech processing system including a first natural-language understanding (NLU) component and a first text-to-speech (TTS) component corresponding to a first speech style indicating output generated by the first speech processing system;

process, using the first NLU component, the first audio data to determine intent data corresponding to an intent associated with the utterance;

cause the device to output light in a first color when processing the first audio data using the first NLU component, wherein processing by a second speech processing system is associated with output of a second color different from the first color;

determine a first ability of the first speech processing system to respond to the intent;

determine a second ability of the second speech processing system to respond to the intent, the second speech processing system including a second TTS component corresponding to a second speech style different from the first speech style and indicating output generated by the second speech processing system;

based at least in part on the second ability and the first ability, causing the second speech processing system to process the intent data;

receive, from the second speech processing system in response to processing the intent data, first response data;

generate, using the first response data and the first TTS component, second audio data corresponding to the first speech style; and

cause the device to output the second audio data.

13. The computing device of claim 12 , wherein the at least one memory further includes instructions that, when executed by the at least one processor, further cause the computing device to:

cause the device to output a first indication associated with the first speech processing system, wherein the second speech processing system is associated with a second indication different from the first indication.

14. The computing device of claim 12 , wherein the at least one memory further includes instructions that, when executed by the at least one processor, further cause the computing device to:

determine that the intent corresponds to a prior command;

determine that a user profile includes second response data corresponding to the prior command and corresponding to the first speech processing system; and

send, to the second speech processing system, the second response data.

15. The computing device of claim 12 , wherein the at least one memory further includes instructions that, when executed by the at least one processor, further cause the computing device to:

send, to the first speech processing system, the intent data;

receive, from the first speech processing system, second response data; and

output, based on the second response data and using the first speech processing system, third audio data.

16. The computing device of claim 12 , wherein the at least one memory further includes instructions that, when executed by the at least one processor, further cause the computing device to:

receive, from the second speech processing system, an indication of an action to be performed by the device, wherein the device is associated with a vehicle; and

cause the device to perform the action, wherein performing the action includes causing a change in a state of the vehicle.

17. The computing device of claim 16 , wherein causing a change in a state of the vehicle includes one or more of raising or lowering a window, setting a thermostat, adjusting a seat, locking or unlocking doors, or turning on or turning off lights.

18. The computing device of claim 12 , wherein the at least one memory further includes instructions that, when executed by the at least one processor, further cause the computing device to:

in response to detecting the first wakeword, cause the device to output audio identifying the first speech processing system.

19. The computing device of claim 12 , wherein the at least one memory further includes instructions that, when executed by the at least one processor, further cause the computing device to:

determine that the first audio data additionally represents a second intent corresponding to the second speech processing system;

process the first audio data using a second NLU component corresponding to the second speech processing system, to determine second intent data corresponding to the second intent;

generate, using the second TTS component, second response data; and

cause the device to output third audio data corresponding to the second response data.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 11, 2018
From: GUPTA, ABHAY; WHALIN, TIMOTHY
To: AMAZON TECHNOLOGIES, INC.
Reel/Frame 047745/0850 →
Continuity (1)
Related Publication 20200184967A1 · Jun 11, 2020
Cited By (1)
US 12,482,465