Granted Patent B1

US 12,625,479 · App. 17/709,759 · Granted May 12, 2026

Inventors: Sven Eberhardt (Seattle, WA); Maisie Wang (Seattle, WA); Kunal Pramod Ghogale (Redmond, WA); Emmett Barton (Seattle, WA); Dustin D Clark (Austin, TX)

Assignee: Amazon Technologies, Inc.

G05B19/042G06F3/14G06N20/00G06V40/161G06V40/168G06V40/172G10L17/02G10L17/06G05B2219/2642

Abstract

Claims (106)

1 . A computing system comprising:

at least one processor; and

at least one memory comprising instructions that, when executed by the at least one processor, cause the computing system to:

determine activity representing at least:

a first usage history indicating a first smart home device performed a first action in response to a first user input received by a first device, and

a second of usage history indicating the first smart home device performed a second action in response to a second user input received by a second device;

based on the activity data, determine a list of smart home devices that performed actions in response to user inputs received by the first device, instead of the second device;

send, to the first device, the list of smart home devices;

in response to receiving the list of smart home devices, determine, by the first device, user data representing characteristics of a user of the first device;

using the user data, determine, by the first device and from the list of smart home devices, one or more smart home devices that performed one or more actions in response to user inputs of the user to the first device;

after determining the one or more smart home devices that performed one or more actions in response to user inputs of the user to the first device, determine, by the first device, occurrence of a trigger event indicating the user is near or engaging with the first device; and

in response to determining occurrence of the trigger event, display, by the first device, a representation of at least one of the one or more smart home devices.

2 . The computing system of claim 1 , wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the computing system to, by the first device:

determine image data using at least one camera of the first device;

process the image data to determine a portion of the image data corresponding to a face;

process the portion of the image data corresponding to the face to determine first facial feature embedding data representing facial features of the face;

determine the user data to be stored facial feature embedding data corresponding to the first facial feature embedding data;

determine usage history data associated with the user data; and

determine the one or more smart home devices that performed one or more actions in response to user inputs of the user to the first device using the list of smart home devices and the usage history data.

3 . The computing system of claim 1 , wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the computing system to, by the first device:

using the user data, determine the user corresponds to a first user type; and

based on determining the user corresponds to the first user type, determine the one or more smart home devices that performed one or more actions in response to user inputs of the user to the first device to omit a third smart home device represented in the list of smart home devices, wherein the third smart home device corresponds to a smart home device type inappropriate to be controlled by a user of the first user type.

4 . The computing system of claim 1 , wherein the trigger event corresponds to one or more of:

determining image data includes a representation of a face;

determining audio data includes speech directed to the first device;

determining the user has navigated to a graphical user interface screen of the first device;

determining a period of time has elapsed;

detecting motion of an object within a vicinity of the first device;

detecting a wireless signal emitted from a third device;

detecting a service set identifier of a wearable device;

detecting a palm the user;

receiving second data representing a height of the user;

receiving third data representing a gait of the user;

receiving fourth data representing at least one of a hair color and a hair length; and

receiving fifth data indicating a fourth device has detected the user.

5 . A computing device comprising:

at least one processor; and

at least one memory comprising instructions that, when executed by the at least one processor, cause the computing device to:

receive a list of content generated based on activity data associated with the computing device;

determine user data representing characteristics of a user of the computing device;

using the user data, determine, from the list of content, one or more instances of content that was generated or output in response to user inputs of the user to the computing device;

after determining the one or more instances of content that was generated or output in response to user inputs of the user to the computing device, determine occurrence of a trigger event indicating the user is near or engaging with the computing device; and

in response to determining occurrence of the trigger event, present an output requesting an input requesting performance of an action corresponding to at least one of the one or more instances of content that was generated or output in response to user inputs of the user to the computing device.

6 . The computing device of claim 5 , wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the computing device to:

determine usage history data associated with the user data; and

determine the one or more instances of content that was generated or output in response to user inputs of the user to the computing device using the list of content and the usage history data.

7 . The computing device of claim 6 , wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the computing device to:

receive image data using at least one camera of the computing device;

perform facial detection processing on the image data to determine a portion of the image data corresponds to a face;

process the portion of the image data corresponding to the face to determine first facial feature embedding data representing facial features of the face;

determine stored facial feature embedding data corresponding to the first facial feature embedding data; and

determine the usage history data to be associated with the stored facial feature embedding data.

8 . The computing device of claim 6 , wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the computing device to:

receive audio data corresponding to a spoken user input;

process the audio data to determine first speech characteristic embedding data representing speech characteristics of the spoken user input;

determine stored speech characteristic embedding data corresponding to the first speech characteristic embedding data; and

determine the usage history data to be associated with the stored speech characteristic embedding data.

9 . The computing device of claim 5 , wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the computing device to:

based on the user data, determine the user corresponds to a first user type; and

determine the one or more instances of content that was generated or output in response to user inputs of the user to the computing device further based on determining the user corresponds to the first user type.

10 . The computing device of claim 5 , wherein the list of content comprises a device name corresponding to a smart home device that is controllable using the computing device.

11 . The computing device of claim 5 , wherein the list of content comprises an identifier corresponding to a group of actions capable of being performed in response to a single user input.

12 . The computing device of claim 5 , wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the computing device to:

receive input data corresponding to a user input, the user input corresponding to the input requesting performance of an action corresponding to at least one of the one or more instances of content that was generated or output in response to user inputs of the user to the computing device; and

commence performance of skill processing in response to the input data.

13 . The computing device of claim 5 , wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the computing device to:

determine a period of time has elapsed since commencing display of first content based on past content that was generated or output in response to user inputs of the user to the computing device; and

based on determining the period of time has elapsed, determine the one or more instances of content that was generated or output in response to user inputs of the user to the computing device and present the output request the input requesting performance of an action corresponding to at least one of the one or more instances of content that was generated or output in response to user inputs of the user to the computing device.

14 . The computing device of claim 5 , wherein:

The computing device corresponds to a first computing device;

the activity data represents at least:

a first usage history indicating first content was generated or output in response to a first user input received by the computing device, and

a second usage history indicating second content was generated or output in response to a second user input received by a second computing device; and

the list of content represents content that was generated or output in response to user inputs received by the first computing device, instead of the second computing device.

15 . A computer-implemented method performed by a computing device, the computer-implemented method comprising:

receiving a list of content generated based on activity data associated with the computing device;

determining user data representing characteristics of a user of the computing device;

using the user data, determining from the list of content, one or more instances of content that was generated or output in response to user inputs of the user to the computing device;

after determining the one or more instances of content that was generated or output in response to user inputs of the user to the computing device, determining occurrence of a trigger event indicating the user is near or engaging with the computing device; and

in response to determining occurrence of the trigger event, presenting an output requesting an input requesting performance of an action corresponding to at least one of the one or more instances of content.

16 . The computer-implemented method of claim 15 , further comprising:

determining usage history data associated with the user data; and

determining the one or more instances of content that was generated or output in response to user inputs of the user to the computing device using the list of content and the usage history data.

17 . The computer-implemented method of claim 16 , further comprising:

receiving image data using at least one camera of the computing device;

performing facial detection processing on the image data to determine a portion of the image data corresponds to a face;

processing the portion of the image data corresponding to the face to determine first facial feature embedding data representing facial features of the face;

determining stored facial feature embedding data corresponding to the first facial feature embedding data; and

determining the usage history data to be associated with the stored facial feature embedding data.

18 . The computer-implemented method of claim 16 , further comprising:

receiving audio data corresponding to a spoken user input;

processing the audio data to determine first speech characteristic embedding data representing speech characteristics of the spoken user input;

determining stored speech characteristic embedding data corresponding to the first speech characteristic embedding data; and

determining the usage history data to be associated with the stored speech characteristic embedding data.

19 . The computer-implemented method of claim 15 , further comprising:

based on the user data, determining the user corresponds to a first user type; and

determining the one or more instances of content that was generated or output in response to user inputs of the user to the computing device further based on determining the user corresponds to the first user type.

20 . The computer-implemented method of claim 15 , wherein the list of content comprises at least one of:

a device name corresponding to a smart home device that is controllable using the computing device; or

an identifier corresponding to a group of actions capable of being performed in response to a single user input.

21 . The computer-implemented method of claim 15 , further comprising:

receiving input data corresponding to a user input, the user input corresponding to the input requesting performance of an action corresponding to at least one of the one or more instances of content that was generated or output in response to user inputs of the user to the computing device; and

commencing performance of skill processing in response to the input data.

22 . The computer-implemented method of claim 15 , further comprising:

determining a period of time has elapsed since commencing display of first content based on past content that was generated or output in response to user inputs of the user to the computing device; and

based on determining the period of time has elapsed, determining the one or more instances of content that was generated or output in response to user inputs of the user to the computing device and presenting the output requesting the input requesting performance of an action corresponding to at least one of the one or more instances of content that was generated or output in response to user inputs of the user to the computing device.

Assignments (1)

References Cited (37)