IP Library › Granted Patent US 10,950,228
Granted Patent B1
US 10,950,228 · App. 15/635,576 · Granted Mar 16, 2021

Interactive voice controlled entertainment

Inventors: Alfred Yong-Hock Tan (Bellevue, WA); Matthew Luker (Port Angeles, WA); David Allen Markley (Bellevue, WA)
Assignee: Amazon Technologies, Inc.
G10L15/22G06F40/279G10L13/00G10L13/08G10L15/26G10L25/57H04N21/2187H04N21/42203H04N21/4302H04N21/4781H04N21/47815H04N21/8126G10L2015/223
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,950,228
App. No.
15/635,576
Granted
Mar 16, 2021
Kind
B1
Abstract

Methods and systems for receiving shouted-out user responses to broadcast entertainment content, and for determining the responsiveness of those responses in relation to the broadcast content. In particular, entertainment broadcasts can be accompanied by mark-up data that represents various events within a given broadcast, which can be compared to the shouted-out responses to determine their accuracy. For example, if a game show was broadcast and an individual started shouting out answers during the broadcast, embodiments disclosed herein could utilize a voice-controlled electronic device that captures the shouted-out answers and passes them on to a language processing system that determines whether they are correct by comparing the answers to the mark-up data. The voice-controlled electronic device can also “listen” to background sounds to capture the broadcast of the entertainment content, and send that content to the language processing system, which can use that captured data to synchronize the actual broadcast with the analysis of the shouted-out answers to provide individuals with an immersive entertainment experience.

Claims (130)

1. A method, comprising:

receiving first audio data representing audio outputted by a first device associated with a display and captured by a voice-controlled device;

determining that the first audio data represents a first portion of content being broadcasted via the first device, the first portion corresponding to a question;

receiving second audio data representing an utterance spoken by a user and captured by the voice-controlled device;

determining that the utterance is in response to the first portion of the content being broadcasted via the first device;

processing the second audio data and the first audio data to determine a response to the utterance; and

generating output data corresponding to the response.

2. The method of claim 1 , further comprising:

receiving a first skill identifier indicating a first skill functionality related to content to be displayed via the first device, wherein the content is interactive content;

determining from the first skill functionality that the voice-controlled device is to operate in a listening mode;

causing the voice-controlled device to activate in the listening mode by:

generating a control message to switch the voice-controlled device from a wakeword detection mode to the listening mode; and

sending the control message to the voice-controlled device enabling the voice-controlled device to capture the first audio data and the second audio data without detecting a wakeword.

3. The method of claim 1 , further comprising:

receiving background audio data representing the content being displayed via the first device;

receiving time synchronization data related to the content that identifies timing of events during broadcast;

processing the background audio data, using automatic speech recognition (ASR), to determine background text data;

processing, using natural language understanding (NLU), the background text data to determine that the background text data matches a second portion of the content being displayed via the first device; and

determining, using the time synchronization data and the second portion of the content, a location within the content that is being broadcast.

4. The method of claim 1 , further comprising:

receiving mark-up data corresponding to the content;

determining that the content is being broadcast live;

receiving supplemental mark-up data during broadcast of the content;

determining updated mark-up data by updating the mark-up data with the supplemental mark-up data; and

wherein processing the second audio data and the first audio data further comprises processing the second audio data, the first audio data and the updated mark-up data to determine the response.

5. The method of claim 1 , further comprising:

receiving commercial mark-up data associated with commercials that have yet to be broadcast during the broadcast of the content;

receiving third audio data representing an input received by the voice-controlled device;

processing the third audio data and the commercial mark-up data to determine that the third audio data is a request to make a purchase associated with the commercial mark-up data;

generating text data to confirm the request to make a purchase;

generating purchase audio data using text-to-speech (TTS) techniques and the text data; and

sending the purchase audio data to the voice-controlled device for playback.

6. The method of claim 1 , further comprising:

associating a first user account with the content;

retrieving, from the first user account, first user account data related to the content;

determining updated first user account data by updating the first user account data after broadcast of the content is complete;

generating user account text data based on the updated first user account data;

applying text-to-speech (TTS) techniques to the user account text data to create user account audio data; and

sending the user account audio data to the voice-controlled device for playback.

7. The method of claim 6 , further comprising:

determining other user account data related to the content;

comparing the other user account data to the first user account data;

determining status of the user associated with the first user account data as compared with other users associated with the other user account data;

determining text data to report a comparison of the user with the other users;

converting the text data to status audio data using TTS techniques; and

sending the status audio data to the voice-controlled device for playback.

8. The method of claim 1 , further comprising:

receiving third audio data captured by the voice-controlled device;

determining that the third audio data represents a second utterance spoken by the user;

processing the third audio data using automatic speech recognition (ASR) to determine input text data;

processing the input text data using natural language understanding (NLU) to determine an intent associated with the second utterance;

determining, based on the intent, that the second utterance is unrelated to the content being displayed via the first device; and

generating second output data corresponding to the intent.

9. The method of claim 1 , further comprising:

receiving a first portion of data corresponding to the content to be broadcasted;

determining, using the first audio data, that the first portion of the data expired; and

receiving, in response to the first portion of the data having expired, a second portion of the data corresponding to the content,

wherein processing the second audio data and the first audio data further comprises processing the second audio data, the first audio data and the second portion of the data to determine the response.

10. The method of claim 1 , further comprising:

receiving input audio data representing a wakeword followed by a second utterance;

process, using automatic speech recognition (ASR), the input audio data to determine input text data;

process the input text data to determine that a wakeword is present;

process the input text data to determine intent of the second utterance; and

generate second output data corresponding to the intent, the second output data being unrelated to the content being displayed via the first device.

11. A language processing system, comprising:

memory;

communications circuitry; and

at least one processor operable to:

receive first audio data representing audio outputted by a first device associated with a display and captured by a voice-controlled device;

determine that the first audio data represents a first portion of content being broadcasted via the first device, the first portion corresponding to a question;

receive second audio data representing an utterance spoken by a user and captured by the voice-controlled device;

determine that the utterance was in response to the first portion of the content being broadcasted via the first device;

process the second audio data and the first audio data to determine a response to the utterance; and

generate output data corresponding to the response.

12. The language processing system of claim 11 , wherein the at least one processor is further operable to:

receive a first skill identifier indicating a first skill functionality related to content to be displayed via the first device, wherein the content is interactive content;

determine from the first skill functionality that the voice-controlled device is to operate in a listening mode;

generate a control message to switch the voice-controlled device from a wakeword detection mode to the listening mode; and

send the control message to the voice-controlled device enabling the voice-controlled device to capture the first audio data and the second audio data without detecting a wakeword.

13. The language processing system of claim 11 , wherein the at least one processor is further operable to:

receive background audio data representing the content being displayed the first device;

receive time synchronization data related to the content that identifies timing of events during broadcast;

process the background audio data, using automatic speech recognition (ASR), to determine background text data;

process, using natural language understanding (NLU), the background text data to determine that the background text data matches a second portion of the content being displayed via the first device; and

determine, using the time synchronization data and the second portion of the content, a location within the content that is being broadcast.

14. The language processing system of claim 11 , wherein the at least one processor is further operable to:

receive mark-up data corresponding to the content;

determine that the content is being broadcast live;

receive supplemental mark-up data during broadcast of the content;

determine updated mark-up data by updating the mark-up data with the supplemental mark-up data; and

process the second audio data, the first audio data and the updated mark-up data to determine the response.

15. The language processing system of claim 11 , wherein the at least one processor is further operable to:

receive commercial mark-up data associated with commercials that have yet to be broadcast during the broadcast of the content;

receive third audio data representing an input received by the first device;

process the third audio data and the commercial mark-up data to determine that the third audio data is a request to make a purchase associated with the commercial mark-up data;

generate text data to confirm the request to make a purchase;

generate purchase audio data using text-to-speech (TTS) techniques and the text data; and

send the purchase audio data to the voice-controlled device for playback.

16. The language processing system of claim 11 , wherein the at least one processor is further operable to:

associate a first user account with the content;

retrieve, from the first user account, first user account data related to the content;

determine updated first user account data by updating the first user account data after broadcast of the content is complete;

generate user account text data based on the updated first user account data;

apply text-to-speech (TTS) techniques to the user account text data to create user account audio data; and

send the user account audio data to the voice-controlled device for playback.

17. The language processing system of claim 16 , wherein the at least one processor is further operable to:

determine other user account data related to the content;

compare the other user account data to the first user account data;

determine status of the user associated with the first user account data as compared with other users associated with the other user account data;

determine text data to report a comparison of the user with the other users;

convert the text data to status audio data using text-to-speech (TTS) techniques; and

send the status audio data to the voice-controlled device for playback.

18. The language processing system of claim 11 , wherein the at least one processor is further operable to:

receive input audio data representing a wakeword followed by a second utterance;

process, using automatic speech recognition (ASR), the input audio data to determine input text data;

process the input text data to determine that a wakeword is present;

process the input text data to determine intent of the second utterance; and

generate second output data corresponding to the intent, the second output data being unrelated to the content being displayed via the first device.

19. The language processing system of claim 11 , wherein the at least one processor is further operable to:

receive a first portion of data corresponding to the content to be broadcasted;

determine, using the first audio data, that the first portion of the data expired;

receive, in response to the first portion of the data having expired, a second portion of the data corresponding to the content; and

process the second audio data, the first audio data and the second portion of the data to determine the response.

20. The language processing system of claim 11 , wherein the at least one processor is further operable to:

receiving third audio data captured by the voice-controlled device;

determining that the third audio data represents a second utterance spoken by the user;

processing the third audio data using automatic speech recognition (ASR) to determine input text data;

processing the input text data using natural language understanding (NLU) to determine an intent associated with the second utterance;

determining, based on the intent, that the second utterance is unrelated to the content being displayed via the first device; and

generating second output data corresponding to the intent.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jul 11, 2017
From: TAN, ALFRED YONG-HOCK; LUKER, MATTHEW; MARKLEY, DAVID ALLEN
To: AMAZON TECHNOLOGIES, INC.
Reel/Frame 042975/0819 →
Cited By (6)
US 12,190,878 US 12,190,879 US 12,198,687 US 12,272,356 US 12,488,800 US 12,505,833