Systems and methods for on-device person recognition and provision of intelligent alerts
The present document describes systems and methods for on-device person recognition and provision of intelligent alerts. The system includes a decentralized multi-camera system for on-device facial recognition. A device (e.g., security camera, video doorbell) captures images/video of a person, processes input image frames, detects face images, filters static faces, and aligns a rotation of the face to be upright and frontal. The device then filters low-quality face images and/or images having a large portion of the face occluded. The device computes a face embedding, compares it against a set of locally stored reference embeddings, and sends matching results to cloud services, which, based on the matching result, notifies the device owner whether the observed person is a known person or a stranger. Face detection and recognition computations are performed on device, not at the cloud. No sensitive information is transmitted off device and privacy is thus preserved.
1 . A method for on-device facial recognition, the method comprising:
storing a library of clusters in a local memory on a wireless network device of a decentralized multi-device system, the wireless network device having a camera, a facial recognition system, and an on-device machine-learning framework, a respective cluster including one or more exemplars that are each associated with a same face, wherein at least one exemplar of the one or more exemplars in a respective cluster includes a reference embedding, an indication of a version of a facial recognition model used by the on-device machine-learning framework to generate the reference embedding, and a quality score of the at least one exemplar in relation to the facial recognition model;
capturing, by the camera on the wireless network device, an image of a person;
detecting, by the wireless network device, a face of the person in the image to provide a face detection;
generating, responsive to the detecting and by the on-device machine-learning framework on the wireless network device, a face embedding representing the face of the person in the image;
verifying, by the facial recognition system on the wireless network device, the face embedding against the clusters stored in the local memory of the wireless network device; and
transmitting a result of the verifying to another device to notify a user of whether the face is a familiar face or an unfamiliar face.
2 . The method of claim 1 , further comprising, after detecting the face in the image and prior to generating the face embedding, transmitting a first update with a session identifier and without a cluster identifier to a cloud service to trigger an alert flow for notifying a device owner of the wireless network device of detection of a familiar face or an unfamiliar face by the wireless network device.
3 . The method of claim 2 , wherein:
the clusters include clusters of face embeddings; and
the method further comprises:
computing a confidence value that measures a recognizability of the face detection;
based on a comparison of the confidence value to a threshold value, determining whether to perform a face embedding computation for the face detection; and
responsive to the confidence value passing the threshold value, performing facial recognition on the face detection by comparing the face embedding to the clusters of face embeddings stored in the local memory of the wireless network device.
4 . The method of claim 3 , further comprising:
identifying, from the clusters stored in the local memory, a matching cluster that matches the face embedding;
inferring an identity of the face in the image based on the identity being associated with the matching cluster; and
transmitting a first notification to the other device to alert the user of the inferred identity of the face in the image.
5 . The method of claim 4 , further comprising updating the local memory by adding the face embedding to the matching cluster for future facial recognition tasks.
6 . The method of claim 4 , further comprising transmitting a second update with the session identifier and the cluster identifier to the cloud service to enable the cloud service to locate a corresponding matching cluster in a reference library associated with the cloud service.
7 . The method of claim 6 , further comprising:
uploading a snapshot of the detected face in the image to a media storage database associated with the cloud service, the snapshot being a cropped version of the image; and
transmitting, to the cloud service, a third update with the session identifier and exemplar information corresponding to the face embedding added to the local memory of the wireless network device.
8 . The method of claim 7 , further comprising, after uploading the snapshot to the media storage database, discarding a local copy of the snapshot and the image.
9 . The method of claim 4 , further comprising:
determining that the clusters of face embeddings stored in the local memory do not include a matching cluster that matches the face embedding; and
transmitting a second notification to the other device to alert the user that the face in the image is unfamiliar.
10 . The method of claim 9 , wherein the second notification includes a request for user input to identify the face in the image.
11 . The method of claim 9 , further comprising, based the determining that the clusters of face embeddings do not include the matching cluster that matches the face embedding:
creating a new cluster for the face embedding, the new cluster including a new cluster identifier; and
adding the face embedding to the new cluster in the local memory.
12 . The method of claim 11 , further comprising:
uploading a snapshot of the face detected in the image to a media storage database associated with the cloud service, the snapshot being a cropped version of the image;
transmitting, to the cloud service, a third update with the session identifier and exemplar information corresponding to the face embedding added to the local memory of the wireless network device; and
after uploading the snapshot to the media storage database, discarding a local copy of the snapshot and the image.
13 . The method of claim 2 , wherein the facial recognition system is a local version of another facial recognition system used at the cloud service.
14 . An electronic device comprising:
a camera configured to capture images or video of a scene;
one or more processors; and
local memory storing:
a local library of reference embeddings organized into clusters, each cluster having one or more exemplars corresponding to a subset of the reference embeddings and being associated with a human face, at least one exemplar of the one or more exemplars in a respective cluster including a reference embedding, an indication of a version of a facial recognition model used to generate the reference embedding, and a quality score of the at least one exemplar in relation to the facial recognition model;
an on-device machine-learning framework configured to, based on a face detection of a face of a person in an image captured by the camera, generate a face embedding representing the face of the person in the image;
a facial recognition system configured to verify the face embedding against the clusters stored in the local memory, the facial recognition system including the facial recognition model used to generate the face embedding; and
executable instructions that, when executed by the one or more processors, cause the one or more processors to:
transmit a result of a verification of the face embedding to another device to notify a user of whether the face is a familiar face or an unfamiliar face.
15 . The electronic device of claim 14 , wherein the instructions further cause the one or more processors to, based on the face detection and prior to generation of the face embedding, transmit a first update with a session identifier and without a cluster identifier to a cloud service to trigger an alert flow for notifying a device owner of the electronic device of detection of a familiar face or an unfamiliar face by the electronic device.
16 . The electronic device of claim 15 , wherein:
the clusters include clusters of face embeddings; and
the instructions further cause the one or more processors to:
compute a confidence value that measures a recognizability of the face detection;
based on a comparison of the confidence value to a threshold value, determine whether to perform a face embedding computation for the face detection; and
responsive to the confidence value passing the threshold value:
perform, based on the comparison of the confidence value to the threshold value, facial recognition on the face detection by comparing the face embedding to the clusters of face embeddings stored in the local memory;
identify, from the clusters stored in the local memory, a matching cluster that matches the face embedding;
infer an identity of the face in the captured image based on the identity being associated with the matching cluster; and
transmit a first notification to the other device to alert the user of the inferred identity of the face in the captured image.
17 . The electronic device of claim 14 , wherein the instructions further cause the one or more processors to:
determine that the clusters of face embeddings stored in the local memory do not include a matching cluster that matches the face embedding; and
transmit a second notification to the other device to alert the user that the face in the captured image is unfamiliar.
18 . The electronic device of claim 17 , wherein the instructions further cause the one or more processors to, based a determination that the clusters of face embeddings do not include the matching cluster that matches the face embedding:
create a new cluster for the face embedding, the new cluster including a new cluster identifier; and
add the face embedding to the new cluster in the local memory.
19 . The electronic device of claim 16 , wherein the recognizability of the face detection measured by the confidence value indicates whether a face embedding subsequently generated from the face detection is more likely or less likely to be similar to embeddings of other face detections of the same person and different from embeddings of face detections of other people.
20 . The electronic device of claim 16 , wherein the confidence value is based on an estimation of an L2-Norm of the face embedding.