IP Library Granted Patent US 11,736,760
Granted Patent B2
US 11,736,760 · App. 17/656,788 · Granted Aug 22, 2023

Video integration with home assistant

Inventors: Jessica Yuan (Union City, CA); James Stewart (San Mateo, CA); Rajeev Nongpiur (Mountain View, CA); Patrick Lister (Mountain View, CA); Chi Yeung Jonathan Ng (San Francisco, CA)
Assignee: Google LLC
H04N21/43615G06Q10/109G10L17/22H04L12/281H04L12/282H04L12/2803H04L67/62H04N21/23418H04N21/25875H04N21/4223
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,736,760
App. No.
17/656,788
Granted
Aug 22, 2023
Kind
B2
Abstract

Methods, devices, systems, and means for video integration with a home assistant device are described herein. The home assistant device interacts with a person in a video stream by capturing, using a network-enabled outdoor video camera, a video stream of an outdoor location of a premises at which the person is present and analyzing the person appearing in the captured video stream to determine an identity of the person. Based on determining the identity of the person, the home assistant device announces a presence of the person that is outdoors and outputs instructions to the person.

Claims (63)

1. A method for using a home assistant device to interact with a person in a video stream, the method comprising:

capturing, using a network-enabled outdoor video camera, a video stream of an outdoor location of a premises at which the person is present;

analyzing the person appearing in the captured video stream to determine an identity of the person;

based on determining the identity of the person, announcing, by the home assistant device, a presence of the person that is outdoors; and

outputting instructions to the person.

2. The method of claim 1 , wherein the analyzing the person appearing in the captured video stream to determine the identity of the person comprises:

identifying the person in the captured video stream.

3. The method of claim 2 , further comprising:

determining that no one is within premises; and

based on determining that no one is within premises, providing a first voice prompt to the identified person.

4. The method of claim 3 , further comprising:

in response to the providing the first voice prompt, recoding a voice message from the identified person; and

sending the voice message and the identity of the person to one or more authorized users of the home assistant device.

5. The method of claim 2 , wherein the outputting the instructions to the identified person comprises:

outputting instructions based on the identity of the identified person.

6. The method of claim 1 , wherein the analyzing the person appearing in the captured video stream to determine the identity of the person comprises:

determining that the person in the captured video stream is unidentified.

7. The method of claim 6 , further comprising:

determining that no one is within premises; and

based on determining that no one is within premises, providing a second voice prompt to the unidentified person.

8. The method of claim 7 , further comprising:

in response to the providing the second voice prompt, recoding a voice message from the unidentified person; and

sending the voice message and an indication of the unidentified person to one or more authorized users of the home assistant device.

9. The method of claim 1 , further comprising:

tracking a location of any item left by the person.

10. The method of claim 9 , wherein the tracking the location of the item comprises:

using video from the network-enabled outdoor video camera to monitor any activity occurring near the item.

11. The method of claim 1 , wherein the outputting the instructions to the person comprises:

outputting the instructions by the network-enabled outdoor video camera; or

outputting the instructions by a smart doorbell.

12. The method of claim 1 , wherein the analyzing the person appearing in the captured video stream to determine the identity of the person comprises:

performing facial recognition to determine if the person matches an authorized user based on stored user data, the stored user data including a name and facial features of the person.

13. The method of claim 1 , wherein the announcing the presence of the person that is outdoors comprises:

announcing the presence of the person based on detecting the person in the video stream; or

announcing the presence of the person based on the person ringing a doorbell.

14. A system for interacting with a person in a video stream, the system comprising:

a network-enabled outdoor video camera that captures a video stream of an outdoor location of a premises and streams the video stream to a cloud-based home assistant integration platform;

the cloud-based home assistant integration platform configured to:

analyze the person appearing in the captured video stream to determine an identity of the person;

a home assistant device comprising a microphone and wireless network interface, the home assistant device configured to:

based on the determination of the identity of the person, announce a presence of the person that is outdoors; and

the network-enabled outdoor video camera configured to:

output instructions to the person.

15. The system of claim 14 , the cloud-based home assistant integration platform configured to:

identify the person in the captured video stream;

determine that no one is within premises; and

generate a first voice prompt; and

the network-enabled outdoor video camera configured to:

output a first voice prompt to the identified person.

16. The system of claim 15 ,

the network-enabled outdoor video camera configured to:

in response to the output of the first voice prompt, record a voice message from the identified person; and

send the voice message and the identity of the person to one or more authorized users of the home assistant device via the cloud-based home assistant integration platform.

17. The system of claim 16 ,

the cloud-based home assistant integration platform configured to:

based on the identity of the person, generate instruction for the identified person; and

the network-enabled outdoor video camera configured to:

output the instructions based on the identity of the identified person.

18. The system of claim 14 , the cloud-based home assistant integration platform configured to:

track a location of any item left by the person.

19. The system of claim 18 , wherein the location of the presence of the item configures the cloud-based home assistant integration platform to:

use video from the network-enabled outdoor video camera to monitor any activity occurring near the item.

20. The system of claim 14 , wherein the network-enabled outdoor video camera is a doorbell.

Assignments (2)
CHANGE OF NAME Recorded Mar 28, 2022
From: GOOGLE INC.
To: GOOGLE LLC
Reel/Frame 059746/0063 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 28, 2022
From: YUAN, JESSICA; STEWART, JAMES; NONGPIUR, RAJEEV; LISTER, PATRICK; NG, CHI YEUNG JONATHAN
To: GOOGLE INC.
Reel/Frame 059415/0447 →
Continuity (4)
Continuation 16877847 · May 19, 2020
Continuation 16230349 · Dec 21, 2018
Continuation 15654491 · Jul 19, 2017
Related Publication 20220217439A1 · Jul 7, 2022