IP Library › Granted Patent US 11,929,844
Granted Patent B2
US 11,929,844 · App. 17/709,077 · Granted Mar 12, 2024

Customized interface based on vocal input

Inventors: Benjamin Brown (San Francisco, CA); Da Huang (Kitchener, CA); Christopher Conover (San Francisco, CA); Lisa Williams (San Francisco, CA); Henry Chung (Waterloo, CA)
Assignee: Google LLC
H04L12/2814G06F16/24573H04L12/2809H04L12/282H04L2012/2847
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,929,844
App. No.
17/709,077
Filed
Mar 30, 2022
Granted
Mar 12, 2024
Kind
B2
Art Unit
2455
USPC
709/223
Abstract

Various arrangements for using captured voice to generate a custom interface controller are presented. A vocal recording from a user may be captured in which a spoken command and multiple smart-home devices are indicated. One or more common functions that map to the multiple smart-home devices may be determined. A custom interface controller may be generated that controls the one or more common functions of each smart-home device of the multiple smart-home devices.

Claims (58)

1. A method for using captured voice to generate a custom interface controller, the method comprising:

capturing, via a mobile device, a vocal recording, wherein the vocal recording comprises a spoken command directed to a plurality of smart-home devices;

transmitting the vocal recording to a cloud-based server system to perform natural language processing on the spoken command, wherein the spoken command indicates one or more locations;

receiving, by an application executed by the mobile device, a response from the cloud-based server system that comprises a plurality of smart-home device identifiers corresponding to the one or more locations, wherein:

the plurality of smart-home device identifiers were identified by the cloud-based server system based on the natural language processing performed on the spoken command as corresponding to the one or more locations;

determining, by accessing a home device function database, a common function that maps to each smart-home device of the plurality of smart-home devices; and

generating, by the application executed by the mobile device, a control element for the one or more locations, which is presented as part of the custom interface controller, that controls only the common function of each smart-home device of the plurality of smart-home devices for which smart-home device identifiers were received from the cloud-based server system.

2. The method for using the captured voice to generate the custom interface controller of claim 1 , further comprising:

receiving, by the application executed by the mobile device, user input via the custom interface controller and a touchscreen of the mobile device.

3. The method for using the captured voice to generate the custom interface controller of claim 2 , further comprising:

determining, by the application executed by the mobile device, a command to be performed by each of the plurality of smart-home devices based on the user input received via the custom interface controller; and

transmitting, by the application executed by the mobile device, the command and the plurality of smart-home device identifiers to the cloud-based server system.

4. The method for using the captured voice to generate the custom interface controller of claim 3 , further comprising:

in response to receiving the command and the plurality of smart-home device identifiers from the application executed by the mobile device, transmitting, to a second cloud-based server system, an indication of the command and a smart-home device identifier of the plurality of smart-home device identifiers, wherein the second cloud-based server system communicates with a second smart-home device mapped to the smart-home device identifier.

5. The method for using the captured voice to generate the custom interface controller of claim 3 , further comprising:

in response to receiving the command and the plurality of smart-home device identifiers from the application executed by the mobile device, transmitting, a second indication of the command to a second smart-home device mapped to a second smart-home device identifier of the plurality of smart-home device identifiers.

6. The method for using the captured voice to generate the custom interface controller of claim 1 , wherein:

the vocal recording additionally indicates a type of smart-home device; and

the cloud-based server system causes the spoken command to be performed at the plurality of smart-home devices based on the plurality of smart-home devices matching the type of smart-home device indicated in the vocal recording.

7. The method for using the captured voice to generate the custom interface controller of claim 1 , further comprising:

analyzing, by the cloud-based server system, the vocal recording to determine the spoken command and the plurality of smart-home devices.

8. The method for using the captured voice to generate the custom interface controller of claim 1 , wherein the plurality of smart-home device identifiers correspond to multiple different types of smart-home devices.

9. A system for using captured voice to generate a custom interface controller, the system comprising a smart-home application executed by a mobile device and a cloud-based host server system, configured to:

capture, via one or more microphones of the mobile device, a vocal recording from a user, wherein the vocal recording comprises a spoken command directed to a plurality of smart-home devices;

transmit, via a network interface of the mobile device, the vocal recording to the cloud-based host server system to perform natural language processing on the spoken command, wherein the spoken command indicates one or more locations;

receive, by the mobile device, a response from the cloud-based host server system that comprises a plurality of smart-home device identifiers corresponding to the one or more locations, wherein:

the plurality of smart-home device identifiers were identified by cloud-based host server system based on the natural language processing performed on the spoken command as corresponding to the one or more locations;

determine, by accessing a home device function database, a common function that maps to each smart-home device of the plurality of smart-home devices; and

generate a control element for the one or more locations, which is presented as part of the custom interface controller, that controls only the common function of each smart-home device of the plurality of smart-home devices for which smart-home device identifiers were received from the cloud-based host server system.

10. The system for using the captured voice to generate the custom interface controller of claim 9 , wherein the smart-home application further causes the mobile device to:

receive, via a touchscreen of the mobile device, user input via the custom interface controller.

11. The system for using the captured voice to generate the custom interface controller of claim 10 , wherein the smart-home application further causes the mobile device to:

determine a command to be performed by each of the plurality of smart-home devices based on the user input received via the custom interface controller; and

transmit the command and the plurality of smart-home device identifiers to the cloud-based host server system.

12. The system for using the captured voice to generate the custom interface controller of claim 11 , wherein the smart-home application further causes the mobile device to:

transmit, to the cloud-based host server system, an indication of the command and a smart-home device identifier of the plurality of smart-home device identifiers in response to receiving the command and the plurality of smart-home device identifiers from the smart-home application executed by the mobile device, wherein the cloud-based host server system communicates with a second smart-home device mapped to the smart-home device identifier.

13. The system for using the captured voice to generate the custom interface controller of claim 12 , wherein the smart-home application further causes the mobile device to:

transmit, a second indication of the command to the second smart-home device mapped to a second smart-home device identifier of the plurality of smart-home device identifiers in response to receiving the command and the plurality of smart-home device identifiers from the smart-home application executed by the mobile device.

14. The system for using the captured voice to generate the custom interface controller of claim 9 , wherein the custom interface controller comprises a first control to control a first function of the plurality of smart-home devices and a second control to control a second function of the plurality of smart-home devices.

15. The system for using the captured voice to generate the custom interface controller of claim 9 , further comprising the cloud-based host server system, wherein:

the vocal recording additionally indicates a type of smart-home device; and

the cloud-based host server system causes the spoken command to be performed at the plurality of smart-home devices based on the plurality of smart-home devices matching the type of smart-home device indicated in the vocal recording.

16. The system for using the captured voice to generate the custom interface controller of claim 9 , further comprising the cloud-based host server system that is configured to analyze the vocal recording to determine the spoken command and the plurality of smart-home devices.

17. The system for using the captured voice to generate the custom interface controller of claim 9 , wherein the plurality of smart-home device identifiers correspond to multiple different types of smart-home device.

18. A system for using captured voice to generate a custom interface controller, the system comprising:

a cloud-based host server system that performs natural language processing;

a mobile device, wherein one or more processors of the cloud-based host server system and the mobile device are configured to:

capture, via one or more microphones, a vocal recording from a user, wherein the vocal recording comprises a spoken command directed to a plurality of smart-home devices;

transmit the vocal recording to the cloud-based host server system to perform the natural language processing on the spoken command, wherein the spoken command indicates one or more locations;

receive a response from the cloud-based host server system that comprises a plurality of smart-home device identifiers corresponding to the one or more locations, wherein:

the plurality of smart-home device identifiers were identified by the cloud-based host server system based on the natural language processing performed on the spoken command as corresponding to the one or more locations;

determine a common function of the plurality of smart-home devices based on the natural language processing performed on the spoken command; and

generate a control element for the one or more locations, which is presented as part of the custom interface controller, for presentation by a display of the mobile device, wherein the control element controls only the common function of each smart-home device of the plurality of smart-home devices for which smart-home device identifiers were received from the cloud-based host server system.

19. The system for using the captured voice to generate the custom interface controller of claim 18 , wherein the one or more processors are further configured to:

receive user input via the custom interface controller;

determine a command to be performed by each of the plurality of smart-home devices based on the user input received via the custom interface controller; and

based on the command, cause an indication of the command and a smart-home device identifier to be transmitted to a second cloud-based host server system distinct from the cloud-based host server system, wherein the second cloud-based host server system communicates with a second smart-home device mapped to the smart-home device identifier.

20. The system for using the captured voice to generate the custom interface controller of claim 18 , wherein the plurality of smart-home device identifiers correspond to multiple different types of smart-home devices.

Continuity (2)
Continuation 16154466 · Oct 8, 2018
Related Publication 20220224557A1 · Jul 14, 2022