IP Library Granted Patent US 11,256,951
Granted Patent B2
US 11,256,951 · App. 16/827,290 · Granted Feb 22, 2022

Systems and methods of person recognition in video streams

Inventors: Akshay R. Bapat (Mountain View, CA); George Alban Heitz, III (Mountain View, CA); Rizwan Ahmed Chaudhry (Palo Alto, CA); Navneet Dalal (Atherton, CA); James Edward Stewart (Mountain View, CA); Jennifer Bush (Mountain View, CA); Joe Delone Venters (Alameda, CA); Kara Gates (Mountain View, CA); Timothy Butler (Mountain View, CA); Yohannes Kifle (San Carlos, CA); Anton Bastov (Mountain View, CA)
Assignee: Google LLC
G06K9/6218G06K9/00362G06K9/00744G08B13/1966G08B13/19645G08B13/19663G08B13/19697H04N7/18G06F16/45G06K9/00228G06K9/00268G06K9/00288G06K2009/00738H04N5/23219
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,256,951
App. No.
16/827,290
Filed
Mar 23, 2020
Granted
Feb 22, 2022
Kind
B2
Art Unit
2663
USPC
382/225
Abstract

The various implementations described herein include systems and methods for recognizing persons in video streams. In one aspect, a method includes: obtaining images collected by video cameras in a smart home environment, each image including a detected person; for each image, obtaining first information that identifies an attribute of the detected person, the first information generated from analysis of the image; grouping the images into a first group of a plurality of groups based on the first information, each group of the plurality of groups representing a unique person; receiving a request to remove a first image from the first group; removing the first image from the first group; and disassociating the first information from the first group.

Claims (71)

1. A method, comprising:

obtaining one or more images collected by one or more video cameras in a smart home environment, each image of the one or more images including a person;

for each image of the one or more images, obtaining information that identifies an attribute of the person, the information generated from analysis of the image;

causing a group of images to be presented, wherein the group of images corresponds to the person, and wherein the information is associated with the group of images;

receiving a request to remove the image from the group of images;

removing the image from the group of images;

generating updated information for the group of images by disassociating the information from the group of images;

after receiving the request, detecting an event captured by a video camera of the one or more video cameras;

determining, based on the updated information, that the event involves the person; and

generating a notification for the event, the notification including an image of the person selected from the group of images.

2. The method of claim 1 , wherein the request to remove the image corresponds to a selection of the image and a subsequent action.

3. The method of claim 1 , wherein detecting the event comprises:

obtaining a new image including the person;

obtaining new information that identifies an attribute of the person, the new information generated from analysis of the new image;

determining that the new image includes the person based on a comparison of the new information with the updated information for the group; and

in accordance with the determination that the new image includes the person:

including the new image in the group; and

associating the new information with the group.

4. The method of claim 1 , further comprising, for each respective group from a plurality of groups, selecting an image of the respective group as a profile image for the respective group, wherein the group of images is included in the plurality of groups.

5. The method of claim 4 , wherein the profile image for a respective group is selected based on a determined mood of a respective person in each image.

6. The method of claim 4 , further comprising displaying a listing of one or more groups from the plurality of groups, the listing including display of the profile image for each of the one or more groups.

7. The method of claim 4 , further comprising displaying an event timeline corresponding to at least one of the one or more video cameras, including:

displaying indication of one or more events on the event timeline; and

for at least one of the one or more events, displaying one or more profile images corresponding to one or more persons in the at least one event.

8. A server system comprising:

one or more processors; and

memory coupled to the one or more processors, the memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for:

obtaining one or more images collected by one or more video cameras in a smart home environment, each image of the one or more images including a person;

for each image of the one or more images, obtaining information that identifies an attribute of the person, the information generated from analysis of the image;

causing a group of images to be presented, wherein the group of images corresponds to the person, and wherein the information is associated with the group of images;

receiving a request to remove the image from the group of images;

removing the image from the group of images;

generating updated information for the group of images by disassociating the information from the group of images;

after receiving the request, detecting an event captured by a video camera of the one or more video cameras;

determining, based on the updated information, that the event involves the person; and

generating a notification for the event, the notification including an image of the person selected from the group of images.

9. The system of claim 8 , wherein the one or more images are grouped into the group in response to a determination that the information corresponding to each image in the one or more images is within a similarity threshold to other information for the group.

10. The system of claim 8 , wherein the one or more programs further comprise instructions for, for a first image of the one or more images, adjusting the information based on a determined pose of the person in the image.

11. The system of claim 8 , wherein the one or more programs further comprise instructions for, adjusting, based on a camera operating mode, the information for at least one image from the one or more images.

12. The system of claim 8 , wherein:

the request to remove the image comprises a request to assign the image to a different group of images from a plurality of groups of images;

removing the image from the group comprises assigning the image to the different group; and

disassociating the information of the image comprises associating the information with the different group.

13. The system of claim 8 , wherein the one or more programs further comprise instructions for:

receiving a request to remove a particular group of images from a plurality of groups of images; and

in response to the request:

deleting all images in the particular group of images; and

deleting corresponding information.

14. The system of claim 8 , wherein the information comprises facial feature information of the person.

15. The system of claim 8 , wherein the information comprises a feature signature for the person, the feature signature generated based on analysis of the image.

16. A non-transitory computer-readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by a computing system, cause the system to:

obtain one or more images collected by one or more video cameras in a smart home environment, each image of the one or more images including a person;

for each image of the one or more images, obtain information that identifies an attribute of the person, the information generated from analysis of the image;

cause a group of images to be presented, wherein the group of images corresponds to the person, and wherein the information is associated with the group of images;

receiving a request to remove the image from the group;

remove the image from the group;

generate updated information for the group by disassociating the information from the group;

after receiving the request, detect an event captured by a first video camera of the one or more video cameras;

determine, based on the updated information, that the event involves the person; and

generate a notification for the event, the notification including an image of the person selected from the group of images.

17. The non-transitory computer-readable storage medium of claim 16 , wherein the one or more programs further comprise instructions, which when executed by the computing system, cause the system to, subsequent to generating the notification:

receive identification information of the person for the group of images; and

remove the group of images from a plurality of groups of images based on the received identification information.

18. The non-transitory computer-readable storage medium of claim 17 , wherein the identification information indicates that the person is a stranger; and

wherein removing the group of images from the plurality of groups of images comprises deleting (i) all images in the group of images, and (ii) the information.

19. The non-transitory computer-readable storage medium of claim 17 , wherein the identification information indicates an identity of the person; and

wherein removing the group of images from the plurality of groups of images comprises combining the group of images with another group of images from the plurality of groups of images representing the same identity.

20. The non-transitory computer-readable storage medium of claim 16 , wherein:

the request to remove the image comprises a request to delete the image;

removing the image from the group of images comprises deleting the image; and

disassociating the information of the image comprises deleting the information.

Assignments (4)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 12, 2022
From: HEITZ, GEORGE ALBAN, III
To: GOOGLE LLC
Reel/Frame 058628/0875 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 8, 2020
From: BAPAT, AKSHAY R.; HEITZ, GEORGE ALBAN; CHAUDHRY, RIZWAN AHMED; DALAL, NAVNEET; STEWART, JAMES EDWARD; BUSH, JENNIFER; VENTERS, JOE DELONE; GATES, KARA; BUTLER, TIMOTHY
To: GOOGLE INC.
Reel/Frame 054572/0097 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 8, 2020
From: KIFLE, YOHANNES; BASTOV, ANTON
To: GOOGLE LLC
Reel/Frame 054572/0154 →
CHANGE OF NAME Recorded Dec 8, 2020
From: GOOGLE INC.
To: GOOGLE LLC
Reel/Frame 054636/0958 →
Continuity (3)
Continuation 15676868 · Aug 14, 2017
Provisional Application 62512697 · May 30, 2017
Related Publication 20200226416A1 · Jul 16, 2020
Cited By (3)
US 12,542,874 US 12,664,818 US 12,682,013