Dynamic gaze correction for video conferencing
View Patent ↗Systems and techniques to dynamically correct a gaze of participants in a video conference are described. A computing device participating in a video conference may determine that a frame of a video stream includes features of a face, extract a portion of the frame that includes a first pair of eyes, and determine that the first pair of eyes are looking in a non-forward direction. The computing device may retrieve, from a database, a stored portion that includes a second pair of eyes that are looking in a forward direction, and modify the frame by substituting the stored portion for the portion in the frame to create a modified frame. The computing device may send the modified frame to a videoconferencing server for distribution to other computing devices participating in the video conference.
1. A computer-implemented method, comprising:
determining, using a classifier, that a first frame of a video stream includes a face;
extracting, using the classifier, a first portion of the first frame that includes a first pair of eyes;
determining, using the classifier, that the first pair of eyes appear to be looking in a non-forward direction, wherein the non-forward direction is relative to a camera;
retrieving, from a database, a stored portion that includes a second pair of eyes that appear to be looking in a forward direction, wherein the forward direction is relative to the camera; and
modifying the first frame by replacing the first portion with the stored portion to create a modified frame;
sending the modified frame to a videoconferencing server;
determining that a second frame of the video stream includes the face;
extracting a second portion of the second frame that includes a second pair of eyes;
determining that the second pair of eyes appear to be looking in the forward direction;
determining that the second portion does not match stored portions in the database;
adding the second portion to the database; and
sending the second frame to the videoconferencing server.
2. The computer-implemented method of claim 1 , further comprising:
determining that a third frame of the video stream excludes the face; and
sending the third frame to the videoconferencing server.
3. The computer-implemented method of claim 1 , wherein retrieving, from the database, the stored portion that includes the second pair of eyes that appear to be looking in the forward direction comprises:
comparing the first portion with one or more stored portions in the database;
determining that, excluding a direction in which the first pair of eyes are looking, the first portion matches the stored portion; and
selecting the stored portion for inclusion in the modified frame.
4. The computer-implemented method of claim 1 , further comprising:
determining a fingerprint comprising a hash of the first portion; and
comparing the fingerprint with a stored fingerprint associated with the stored portion.
5. The computer-implemented method of claim 1 , further comprising:
capturing one or more frames from the camera during a particular period of time;
determining that a particular frame of the one or more frames includes features of the face;
extracting, from the particular frame, a particular portion that includes the second pair of eyes; and
storing the particular portion in the database to create the stored portion.
6. The computer-implemented method of claim 5 , further comprising:
displaying a prompt requesting that a participant look in a direction of a camera for the particular period of time.
7. The computer-implemented method of claim 1 , wherein modifying the first frame by replacing the first portion with the stored portion to create the modified frame comprises:
performing image stitching to overlay the stored portion over the first portion to create the modified frame.
8. One or more non-transitory computer-readable media storing instructions that are executable by one or more processors to perform operations comprising:
determining, using a classifier, that a first frame of a video stream includes a face;
extracting, using the classifier, a first portion of the first frame that includes a first pair of eyes;
determining, using the classifier, that the first pair of eyes appear to be looking in a non-forward direction relative to a camera;
retrieving, from a database, a stored portion that includes a second pair of eyes that appear to be looking in a forward direction relative to the camera; and
modifying the first frame by replacing the first portion with the stored portion to create a modified frame;
sending the modified frame to a videoconferencing server;
determining that a second frame of the video stream includes the face;
extracting a second portion of the second frame that includes a second pair of eyes;
determining that the second pair of eyes appear to be looking in the forward direction;
determining that the second portion does not match stored portions in the database;
adding the second portion to the database; and
sending the second frame to the videoconferencing server.
9. The one or more non-transitory computer-readable media of claim 8 , wherein retrieving, from the database, the stored portion that includes the second pair of eyes that appear to be looking in the forward direction comprises:
comparing the first portion with one or more stored portions in the database;
determining that, excluding a direction in which the first pair of eyes are looking, the first portion matches the stored portion; and
selecting the stored portion for inclusion in the modified frame.
10. The one or more non-transitory computer-readable media of claim 9 , the operations further comprising:
determining one or more features associated with the first portion; and
comparing the one or more features with particular features associated with the stored portion.
11. The one or more non-transitory computer-readable media of claim 9 , the operations further comprising:
determining a fingerprint comprising a hash of the first portion; and
comparing the fingerprint with a stored fingerprint associated with the stored portion.
12. The one or more non-transitory computer-readable media of claim 8 , the operations further comprising:
displaying a prompt requesting that a participant look in a direction of a camera for a particular period of time; and
capturing one or more frames from the camera.
13. The one or more non-transitory computer-readable media of claim 8 , the operations further comprising:
creating the modified frame by using image stitching to overlay the stored portion over the first portion.
14. A computing device, comprising:
one or more processors; and
one or more non-transitory computer-readable media storing instructions that are executable by the one or more processors to perform operations comprising:
determining, using a classifier, that a first frame of a video stream includes a face;
extracting, using the classifier, a first portion of the first frame that includes a first pair of eyes;
determining, using the classifier, that the first pair of eyes appear to be looking in a non-forward direction;
retrieving, from a database, a stored portion that includes a second pair of eyes that appear to be looking in a forward direction; and
modifying the first frame by substituting the first portion with the stored portion to create a modified frame;
sending the modified frame to a videoconferencing server;
determining that a second frame of the video stream includes the face;
extracting a second portion of the second frame that includes a second pair of eyes;
determining that the second pair of eyes appear to be looking in a forward direction; and
sending the second frame to the videoconferencing server;
determining that the second portion does not match stored portions in the database; and
adding the second portion to the database.
15. The computing device of claim 14 , the operations further comprising:
determining one or more features associated with the first portion; and
comparing the one or more features with particular features associated with the stored portion.
16. The computing device of claim 14 , the operations further comprising:
determining that a third frame of the video stream excludes the face; and
sending the third frame to the videoconferencing server.
17. The computing device of claim 14 , the operations further comprising:
capturing one or more frames during a particular period of time;
determining that a particular frame of the one or more frames includes the face;
extracting, from the particular frame, a particular portion that includes the second pair of eyes; and
storing the particular portion in the database to create the stored portion.
18. The computing device of claim 14 , the operations further comprising:
performing image stitching to overlay the stored portion over the first portion to create the modified frame.
19. The computing device of claim 14 , the operations wherein retrieving, from the database, the stored portion that includes the second pair of eyes that appear to be looking in the forward direction comprises:
comparing the first portion with one or more stored portions in the database;
determining that, excluding a direction in which the first pair of eyes are looking, the first portion matches the stored portion; and
selecting the stored portion for inclusion in the modified frame.
20. The computing device of claim 19 , the operations further comprising:
determining a fingerprint comprising a hash of the first portion; and
comparing the fingerprint with a stored fingerprint associated with the stored portion.