IP Library Granted Patent US 10,687,109
Granted Patent B2
US 10,687,109 · App. 16/230,349 · Granted Jun 16, 2020

Video integration with home assistant

Inventors: Jessica Yuan (Union City, CA); James Stewart (San Mateo, CA); Rajeev Nongpiur (Mountain View, CA); Patrick Lister (Mountain View, CA); Chi Yeung Jonathan Ng (San Francisco, CA)
Assignee: Google LLC
H04N21/43615G06Q10/109G10L17/22H04L12/281H04L12/282H04L12/2803H04L67/325H04N21/23418H04N21/25875H04N21/4223
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,687,109
App. No.
16/230,349
Granted
Jun 16, 2020
Kind
B2
Abstract

Various arrangements are detailed herein related to modifying a shopping list using a home assistant device and an integrated video service. A network-enabled video camera may be used to capture a video stream of a location at which a user is present. A home assistant device may receive a spoken command that requests an item be added to a shopping list. The video stream captured using the network-enabled video camera may be analyzed to determine an identity of the user. A shopping list to modify may be determined based on the identity of the user. The determined shopping list may then be modified to indicate the item requested in the spoken command.

Claims (48)

1. A method for modifying a shopping list using a home assistant device and an integrated video service, the method comprising:

capturing, using a network-enabled video camera, a video stream of a location at which a user is present;

receiving, by a home assistant device, a spoken command that requests an item be added to a shopping list;

analyzing the video stream captured using the network-enabled video camera to determine an identity of the user;

determining the shopping list to modify based on the identity of the user; and

modifying the determined shopping list to indicate the item requested in the spoken command.

2. The method for modifying the shopping list using the home assistant device and the integrated video service of claim 1 , wherein the spoken command requires a visual reference to determine the item to be added to the shopping list.

3. The method for modifying the shopping list using the home assistant device and the integrated video service of claim 2 , the method further comprising:

in response to the spoken command requiring the visual reference to determine the item to be added to the shopping list, analyzing the video stream to determine an item being referenced by the user; and

identifying the item based on analyzing the video stream.

4. The method for modifying the shopping list using the home assistant device and the integrated video service of claim 2 , wherein the item being referenced by the user is being held by the user or being pointed to by the user.

5. The method for modifying the shopping list using the home assistant device and the integrated video service of claim 1 , wherein determining the shopping list to modify is further based on the item to be added to the shopping list.

6. The method for modifying the shopping list using the home assistant device and the integrated video service of claim 1 , further comprising:

streaming, by the network-enabled video camera, the video stream to a cloud-based home assistant integration platform.

7. The method for modifying the shopping list using the home assistant device and the integrated video service of claim 6 , further comprising:

storing, by the cloud-based home assistant integration platform, the video stream for at least a period of time.

8. The method for modifying the shopping list using the home assistant device and the integrated video service of claim 7 , wherein analyzing the video stream captured using the network-enabled video camera to determine the identity of the user is performed by the cloud-based home assistant integration platform.

9. The method for modifying the shopping list using the home assistant device and the integrated video service of claim 8 , further comprising:

transmitting, by the home assistant device, the spoken command that requests the item be added to the shopping list to the cloud-based home assistant integration platform; and

analyzing, by the cloud-based home assistant integration platform, the spoken command using a speech recognition engine and a natural language interpretation engine.

10. A system for managing a shopping list, the system comprising:

a network-enabled video camera that captures a video stream of a location and streams the video stream to a cloud-based home assistant integration platform;

a home assistant device comprising a microphone and wireless network interface, the home assistant device configured to:

receive, via the microphone, a spoken command that requests an item be added to a shopping list; and

the cloud-based home assistant integration platform, configured to:

analyze the video stream captured using the network-enabled video camera to determine an identity of a user;

determine the shopping list to modify based on the identity of the user; and

modify the determined shopping list to indicate the item requested in the spoken command.

11. The system for managing the shopping list of claim 10 , wherein the spoken command requires a visual reference to determine the item to be added to the shopping list.

12. The system for managing the shopping list of claim 11 , wherein cloud-based home assistant integration platform is further configured to:

analyze the video stream to determine an item being referenced by the user in response to the spoken command requiring the visual reference to determine the item to be added to the shopping list; and

identify the item based on analyzing the video stream.

13. The system for managing the shopping list of claim 11 , wherein the item being referenced by the user is being held by the user or being pointed to by the user.

14. The system for managing the shopping list of claim 10 , wherein the cloud-based home assistant integration platform being configured to determine the shopping list to modify is further based on the item to be added to the shopping list.

15. The system for managing the shopping list of claim 10 , wherein the network-enabled video camera is further configured to stream the video stream to the cloud-based home assistant integration platform via the wireless network interface.

16. The system for managing the shopping list of claim 15 , wherein the cloud-based home assistant integration platform is further configured to store the video stream for at least a period of time.

17. The system for managing the shopping list of claim 10 , wherein:

the home assistant device is further configured to transmit the spoken command that requests the item be added to the shopping list to the cloud-based home assistant integration platform via the wireless network interface; and

the cloud-based home assistant integration platform is further configured to analyze the spoken command using a speech recognition engine and a natural language interpretation engine.

18. A non-transitory processor-readable medium comprising processor-readable instructions configured to cause one or more processors to:

receive a video stream of a location at which a user is present;

receive a spoken command that requests an item be added to a shopping list;

analyze the video stream captured using a network-enabled video camera to determine an identity of the user;

determine the shopping list to modify based on the identity of the user; and

modify the determined shopping list to indicate the item requested in the spoken command.

19. The non-transitory processor-readable medium of claim 18 , wherein the processor-readable instructions, when executed, further cause the one or more processors to:

analyze the video stream to determine an item being referenced by the user in response to the spoken command; and

identify the item based on analyzing the video stream.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 30, 2021
From: YUAN, JESSICA; STEWART, JAMES; NONGPIUR, RAJEEV; LISTER, PATRICK; NG, CHI YEUNG JONATHAN
To: GOOGLE INC.
Reel/Frame 058509/0056 →
CHANGE OF NAME Recorded Dec 30, 2021
From: GOOGLE INC.
To: GOOGLE LLC
Reel/Frame 058601/0258 →