IP Library Granted Patent US 10,200,746
Granted Patent B1
US 10,200,746 · App. 15/654,491 · Granted Feb 5, 2019

Video integration with home assistant

Inventors: Jessica Yuan (Union City, CA); James Stewart (San Mateo, CA); Rajeev Nongpiur (Mountain View, CA); Patrick Lister (Mountain View, CA); Chi Yeung Jonathan Ng (San Francisco, CA)
Assignee: Google LLC
H04N21/43615G10L17/22H04L12/281H04L12/282H04L12/2803H04L67/325
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,200,746
App. No.
15/654,491
Filed
Jul 19, 2017
Granted
Feb 5, 2019
Kind
B1
Art Unit
2658
USPC
704/275
Abstract

Various arrangements are detailed herein related to managing scheduled events using a home assistant and an integrated video service. Using a network-enabled video camera, a video stream of a location may be captured. The video stream captured may be used to determine an identity of a user present at the location. In response to identifying the identity of the user, access to a cloud-based calendar account for the user may be determined to be authorized. The cloud-based calendar account linked with the user may be accessed. A next calendar event mapped to the cloud-based calendar account that is scheduled to occur within a threshold time duration may be determined. A description linked with the next calendar event may be accessed. A network-enabled smart home assistant device may be used to synthesize speech to announce details related to the next event.

Claims (103)

1. A method for managing scheduled events using a home assistant and an integrated video service, the method comprising:

capturing, using a network-enabled video camera, a video stream of a location at which a user is present;

analyzing the video stream captured using the network-enabled video camera to determine an identity of the user;

in response to identifying the identity of the user, determining that access to a cloud-based calendar account for the user is authorized;

accessing the cloud-based calendar account linked with the user;

determining that a next calendar event mapped to the cloud-based calendar account is scheduled to occur within a threshold time duration;

in response to determining that the next calendar event mapped to the cloud-based calendar account is scheduled to occur within the threshold time duration, accessing a description linked with the next calendar event; and

synthesizing speech, using a network-enabled smart home assistant device, to announce a scheduled start time and read a description linked with the next calendar event while the user remains present within the video stream.

2. The method for managing scheduled events using the home assistant of claim 1 , further comprising:

determining that an event location is linked with the next calendar event;

accessing a traffic service to determine a travel time from a home location linked with the network-enabled smart home assistant device or the network-enabled video camera to the event location; and

synthesizing speech, using the network-enabled smart home assistant device, that indicates a time at which the user should leave the home location to reach the event location based on the determined travel time.

3. The method for managing scheduled events using the home assistant of claim 1 , further comprising:

receiving, by the network-enabled smart home assistant device, a spoken command that requests a video clip be recorded; and

storing, to a cloud-based video storage platform, the video clip from the video stream, wherein the video clip is mapped to a user account of the user.

4. The method for managing scheduled events using the home assistant of claim 3 , further comprising:

receiving, by the network-enabled smart home assistant device, a spoken command that requests the video clip be recorded at a time in the future;

scheduling, by the network-enabled smart home assistant device, a recording event based on the spoken command and the time; and

storing, to the cloud-based video storage platform, the video clip from the video stream starting at the time, wherein the video clip is mapped to a user account of the user.

5. The method for managing scheduled events using the home assistant of claim 1 , further comprising:

receiving, by the network-enabled smart home assistant device, a spoken command that requests an image be captured;

synthesizing speech, using the network-enabled smart home assistant device, indicating an amount of time until the image is captured;

capturing the image using the network-enabled video camera; and

storing, to a cloud-based video storage platform, the image, wherein the image is mapped to a user account of the user.

6. The method for managing scheduled events using the home assistant of claim 1 , further comprising:

receiving, by the network-enabled smart home assistant device, a spoken command that requests a time-lapse video clip be recorded, the spoken command specifying a start time and an end time;

capturing, using the network-enabled video camera, a plurality of images between the start time and the end time, wherein a frequency of images captured is dependent on a duration of time between the start time and the end time; and

creating the time-lapse video clip using the plurality of images.

7. The method for managing scheduled events using the home assistant of claim 1 , further comprising:

receiving, by the network-enabled smart home assistant device, a spoken command that requests a history of one or more events at the location;

analyzing a stored video stream for at least a predefined time period;

identifying a plurality of events linked with an audio level above a defined threshold; and

creating a video highlight reel that includes video from the stored video stream recorded during the plurality of events.

8. The method for managing scheduled events using the home assistant of claim 1 , further comprising:

receiving a first spoken query from an identified user;

determining the user is authorized to access the cloud-based calendar account linked with the network-enabled smart home assistant device; and

synthesizing speech, by the network-enabled smart home assistant device, to announce a response to the first spoken query that incorporates information from the cloud-based calendar account linked with the network-enabled smart home assistant device.

9. The method for managing scheduled events using the home assistant of claim 8 , further comprising:

analyzing the video stream captured using the network-enabled video camera to determine an unidentified user is present;

receiving a second spoken query from the unidentified user;

determining the unidentified user is not authorized to access any calendar linked with the network-enabled smart home assistant device; and

synthesizing speech, by the network-enabled smart home assistant device, to announce a generic response to the second spoken query that does not incorporate information from any calendar linked with the network-enabled smart home assistant device.

10. The method for managing scheduled events using the home assistant of claim 1 , further comprising:

receiving a spoken query from an identified user;

determining that the spoken query is requesting an item be added to a stored shopping list;

in response to determining the spoken query, determining a name of the item by performing object recognition on one or more frames of the video stream; and

adding the determined name of the item to the stored shopping list.

11. A system for managing scheduled events, the system comprising:

a network-connected video camera that captures a video stream of a location; and

a cloud-based home assistant integration platform, configured to:

analyze the video stream captured using the network-connected video camera to determine an identity of a user present in the video stream;

in response to identifying the identity of the user, determine that access to a cloud-based calendar account for the user is authorized;

access the cloud-based calendar account linked with the user from a plurality of cloud-based calendar accounts accessible by the cloud-based home assistant integration platform;

determine that a next calendar event mapped to the cloud-based calendar account is scheduled to occur within a threshold time duration;

in response to determining that the next calendar event mapped to the cloud-based calendar account is scheduled to occur within the threshold time duration, access a description linked with the next calendar event; and

transmit an indication of the next calendar event and the accessed description; and

a home assistant device that is configured to:

receive the indication of the next calendar event and the accessed description; and

synthesize speech to announce the next calendar event and the accessed description.

12. The system for managing scheduled events of claim 11 , wherein the home assistant device and the network-connected video camera are incorporated as part of a single device.

13. The system for managing scheduled events of claim 11 , wherein:

the cloud-based home assistant integration platform is further configured to:

determine an event location linked with the next calendar event;

access a traffic service to determine a travel time from a home location to the event location; and

transfer an indication of the travel time to the home assistant device; and

the home assistant device is further configured to:

synthesize speech that indicates a time at which the user should leave the home location to reach the event location based on the determined travel time or the travel time.

14. The system for managing scheduled events of claim 11 , wherein:

the cloud-based home assistant integration platform is configured to:

receive a spoken command, via the home assistant device, that requests a video clip be recorded; and

store, to a cloud-based video storage arrangement, the video clip from the video stream, wherein the video clip is mapped to a user account of the user who spoke the command to the home assistant device.

15. The system for managing scheduled events of claim 14 , wherein:

the cloud-based home assistant integration platform is configured to:

receive, via the home assistant device, as part of the spoken command, a request that the video clip be recorded at a time in the future; and

schedule a recording event based on the spoken command and the time;

and wherein the cloud-based home assistant integration platform being configured to store the video clip comprises the cloud-based home assistant integration platform storing the video clip in response to the recording event.

16. The system for managing scheduled events of claim 11 , wherein:

the cloud-based home assistant integration platform is configured to:

receive, via the network-connected smart home assistant device, a spoken command that requests an image be captured;

cause the home assistant device to synthesize speech indicating an amount of time until the image is captured; and

store the image, wherein the image is mapped to a user account of the user that spoke the spoken command; and

the network-connected video camera is further configured to capture the image.

17. The system for managing scheduled events of claim 11 , wherein:

the cloud-based home assistant integration platform is further configured to:

analyze the video stream captured using the network-connected video camera to determine an unidentified user is present;

receive a second spoken query from the unidentified user;

determine the unidentified user is not authorized to access any calendar linked with the home assistant device; and

cause the smart home assistant device to synthesize speech indicating a generic response to the second spoken query that does not incorporate information from any users' calendars linked with the home assistant device.

18. A non-transitory processor-readable medium comprising processor-readable instructions configured to cause one or more processors to:

receive, from a network-enabled video camera, a video stream of a location at which a user is present;

analyze the video stream captured using the network-enabled video camera to determine an identity of the user;

in response to identifying the identity of the user, determine that a cloud-based calendar account for the user is accessible;

access the cloud-based calendar account linked with the user;

determine that a next calendar event mapped to the cloud-based calendar account is scheduled to occur within a threshold time duration;

in response to determining that the next calendar event mapped to the cloud-based calendar account is scheduled to occur within the threshold time duration, access a description linked with the next calendar event; and

cause a network-enabled smart home assistant device to output synthesized speech indicating a scheduled start time and read a description linked with the next calendar event.

19. The non-transitory processor-readable medium of claim 18 , wherein the processor-readable instructions, when executed, further cause the one or more processors to:

determine that an event location is linked with the next calendar event;

access a traffic service to determine a travel time from a home location linked with the network-enabled smart home assistant device or the network-enabled video camera to the event location; and

cause the network-enabled smart home assistant device to output synthesized speech that indicates a time at which the user should leave the home location to reach the event location based on the determined travel time.

20. The non-transitory processor-readable medium of claim 18 , wherein the processor-readable instructions, when executed, further cause the one or more processors to:

receive a spoken command that requests a video clip be recorded; and

store, to a cloud-based video storage platform, the video clip from the video stream, wherein the video clip is mapped to a user account of the user that spoke the spoken command.

Assignments (2)
CHANGE OF NAME Recorded Oct 20, 2017
From: GOOGLE INC.
To: GOOGLE LLC
Reel/Frame 044567/0001 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jul 19, 2017
From: YUAN, JESSICA; STEWART, JAMES; LISTER, PATRICK; NG, CHI YEUNG JONATHAN; NONGPIUR, RAJEEV
To: GOOGLE INC.
Reel/Frame 043048/0227 →