Smart cameras enabled by assistant systems
In one embodiment, a method includes accessing sensory data captured by cameras, identifying people in a field of view of the cameras based on facial recognition of the sensory data, detecting actions of one or more of the people based on the sensory data, generating media files with each being associated with one or more of a recording of at least one of the people or at least one of the determined actions, and sending instructions for presenting one or more of the media files to a client system.
1 . A method comprising, by one or more computing systems:
accessing sensory data comprising visual data captured by one or more cameras of head-mounted display configured to be worn by a user;
using the visual data, detecting at least two people in a field of view of the one or more cameras and further detecting facial expressions associated with each of the at least two people;
determining, using one or more machine-learning models, whether each respective person of the at least two people is a point of interest of the visual data;
identifying a person of interest from the at least two people as a point of interest of the visual data;
in accordance with determining that a first privacy setting is associated with the person of interest, the first privacy setting indicating that the user is allowed to identify the person of interest within a respective media sharing application:
adjusting capture of the visual data based at least in part on the person of interest; and
generating a media file comprising a recording of the person of interest;
identifying another person of interest from the at least two people as another point of interest of the visual data; and
in accordance with determining that a second privacy setting is associated with the other person of interest, the second privacy setting indicating that the user is not allowed to identify the other person of interest within the respective media sharing application:
forgoing generating any media files comprising any recordings of the other person of interest.
2 . The method of claim 1 , wherein the one or more machine-learning models comprise a facial recognition algorithm, and identifying the person of interest comprises:
determining, based on providing the visual data as an input to the facial recognition algorithm, one or more identifiers of the at least two people; and
determining an association in a social graph between the user and a particular identifier corresponding to the person of interest,
wherein the association in the social graph between the user and the particular identifier indicates that the person of interest has a relationship with the user.
3 . The method of claim 2 , wherein identifying the person of interest further comprises:
determining measures related to the point of interest for each of the one or more identifiers of the at least two people, wherein the point of interest is determined by providing the social graph and the one or more identifiers as inputs to one or more of the one or more machine-learning models; and
based on an output of the one or more machine-learning models, further determining that the particular identifier corresponds to the point of interest more than identifiers corresponding to other detected people.
4 . The method of claim 1 , wherein the one or more machine-learning models comprise a facial expression recognition algorithm, and identifying the person of interest comprises:
determining, based on providing the visual data as an input to the facial expression recognition algorithm, the facial expressions associated with each of the at least two people; and
determining that the person of interest has a particular facial expression.
5 . The method of claim 1 , wherein the sensory data further comprises eye gaze data of the user captured by the head-mounted display, and identifying the person of interest comprises using the eye gaze data to determine a respective point of interest that is looking at.
6 . The method of claim 1 , further comprising:
detecting one or more objects in the field of view; and
identifying, using the visual data as an input to one or more machine-learning models, an object of interest among the detected objects in the field of view,
wherein the media file comprises a recording of the object of interest.
7 . The method of claim 1 , further comprising:
receiving, from the head-mounted display, a user input comprising one of a text input, an audio input, an eye gaze, a gesture, and a motion, wherein identifying the person of interest is responsive to the user input.
8 . The method of claim 1 , wherein the media file comprises at least one of an image and or a video clip.
9 . The method of claim 1 , further comprising:
generating, based on a plurality of media files, one or more highlight files,
wherein each highlight file comprises a respective media file that satisfies a predefined quality standard, wherein each highlight file is associated with a respective captioning, and wherein one or more presented media files comprise the one or more highlight files.
10 . The method of claim 9 , wherein the predefined quality standard is based on one or more of blurriness, lighting, or vividness of color.
11 . The method of claim 1 , further comprising:
receiving, from the client system, a user query from a user in response to the presented media files;
accessing a plurality of episodic memories associated with the user;
identifying one or more episodic memories of the accessed episodic memories as related to the user query;
retrieving one or more media files corresponding to the identified episodic memories, wherein each media file comprises one or more of a post, a comment, an image, or a video clip; and
sending, to the client system, instructions for presenting the one or more retrieved media files corresponding to the identified episodic memories.
12 . The method of claim 1 , further comprising:
sending, to the head-mounted display, instructions for zooming in one or more of the cameras to position the point of interest in a center of the field of view.
13 . The method of claim 1 , further comprising:
sending, to the client system, instructions for zooming out one or more of the cameras to position one or more of the identified of interest in a center of the field of view.
14 . The method of claim 1 , wherein the presented media files are personalized for a user associated with the client system based on one or more of:
user profile data associated with the user;
user preferences associated with the user;
prior user inputs by the user; or
user relationships with other users in a social graph.
15 . The method of claim 1 , wherein the sensory data further comprises audio signals captured by one or more microphones of the head-mounted display, and identifying the person of interest comprises using the audio signals as a further input to the one or more machine-learning models.
16 . One or more computer-readable non-transitory non-volatile storage media embodying software that is operable when executed to:
access sensory data comprising visual data captured by one or more cameras of a head-mounted display configured to be worn by a user;
using the visual data, detect at least two people in a field of view of the one or more
cameras and further detect facial expressions associated with each of the at least two people;
determine, using one or more machine-learning models, whether each respective person of the at least two people is a point of interest of the visual data;
identify a person of interest from the at least two people as a point of interest of the visual data;
in accordance with determining that a first privacy setting is associated with the person of interest, the first privacy setting indicating that the user is allowed to identify the person of interest within a respective media sharing application:
adjust capture of the visual data based at least in part on the person of interest; and
generate a media file comprising a recording of the person of interest;
identify another person of interest from the at least two people as another point of interest of the visual data; and
in accordance with determining that a second privacy setting is associated with the other person of interest, the second privacy setting indicating that the user is not allowed to identify the other person of interest within the respective media sharing application:
forgoing generating any media files comprising any recordings of the other person of interest.
17 . The one or more computer-readable non-transitory non-volatile storage media of claim 16 , wherein the one or more machine-learning models comprise a facial recognition algorithm, and identifying the person of interest comprises:
determining, based on providing the visual data as an input to the facial recognition algorithm, one or more identifiers of the at least two people; and
determining an association in a social graph between the user and a particular identifier corresponding to the person of interest,
wherein the association in the social graph between the user and the particular identifier indicates that the person of interest has a relationship with the user.
18 . The one or more computer-readable non-transitory non-volatile storage media of claim 16 , wherein identifying the person of interest further comprises:
determining measures related to the point of interest for each of the one or more identifiers of the at least two people, wherein the point of interest is determined by providing the social graph and the one or more identifiers as inputs to one or more of the one or more machine-learning models; and
based on an output of the one or more machine-learning models, further determining that the particular identifier corresponds to the point of interest more than identifiers corresponding to other detected people.
19 . A system comprising:
one or more processors; and
a non-transitory non-volatile memory coupled to the processors comprising instructions executable by the processors, the processors operable when executing the instructions to:
access sensory data comprising visual data captured by one or more cameras of a head-mounted display configured to be worn by a user;
using the visual data, detect at least two people in a field of view of the one or more cameras and further detect facial expressions associated with each of the at least two people;
determine, using one or more machine-learning models, whether each respective person of the at least two people is a point of interest of the visual data;
identify a person of interest from the at least two people as a point of interest of the visual data;
in accordance with determining that a first privacy setting is associated with the person of interest, the first privacy setting indicating that the user is allowed to identify the person of interest within a respective media sharing application:
adjust capture of the visual data based at least in part on the person of interest; and
generate a media file comprising a recording of the person of interest;
identify another person of interest from the at least two people as another point of interest of the visual data; and
in accordance with determining that a second privacy setting is associated with the other person of interest, the second privacy setting indicating that the user is not allowed to identify the other person of interest within the respective media sharing application:
forgoing generating any media files comprising any recordings of the other person of interest.
20 . The system of claim 19 , wherein the one or more machine-learning models comprise a facial recognition algorithm, and identifying the person of interest comprises:
determining, based on providing the visual data as an input to the facial recognition algorithm, one or more identifiers of the at least two people; and
determining an association in a social graph between the user and a particular identifier corresponding to the person of interest,
wherein the association in the social graph between the user and the particular identifier indicates that the person of interest has a relationship with the user.