Efficient image capturing based on eyelid position
Systems, methods, and computer programs for eye tracking involving a camera that is operated according to eyelid position. In one embodiment, an inward-facing head-mounted camera captures images of an eye. A head-mounted device emits electromagnetic waves towards the eye and measures reflections of these waves from the eye. The average rate of reflection measurements is at least ten times the average rate of image capture. A computer detects the position of at least one of the eyelids covering the eye based on the reflection measurements. And the camera operation is adjusted according to the eyelid position in a manner that decreases, in the captured images, areas depicting the eyelids, and optionally increases areas depicting the iris.
1. An eye tracking system configured to operate a camera according to eyelid position, comprising:
an inward-facing head-mounted camera configured to capture images of an eye;
a head-mounted device configured to emit electromagnetic waves towards the eye, and to take measurements of reflections of the electromagnetic waves from the eye; wherein an average rate at which the measurements of the reflections are taken is higher than an average rate at which the images are captured; and
a computer configured to:
detect, based on the measurements of the reflections, a position of at least one of the eyelids covering the eye (eyelid position); and
operate the inward-facing head-mounted camera according to the eyelid position in a manner that decreases, in the captured images, areas depicting the at least one of the eyelids.
2. The eye tracking system of claim 1 , wherein the inward-facing head-mounted camera utilizes a sensor that supports changing of its region of interest (ROI), and the computer is further configured to utilize the eyelid position to set the ROI around a portion of the eye that is not covered by the at least one of the eyelids.
3. The eye tracking system of claim 2 , wherein the ROI covers less than half the sensor's field of view, and wherein the average rate at which the measurements of the reflections are taken is at least ten times the average rate at which the images are captured.
4. The eye tracking system of claim 2 , wherein the computer is configured to set the ROI such that the size of the ROI is proportional to area of the eyeball uncovered by the eyelid.
5. The eye tracking system of claim 2 , wherein the computer is further configured to operate the inward-facing head-mounted camera according to the eyelid position in a manner that increases, in the captured images, areas depicting the iris.
6. The eye tracking system of claim 2 , wherein the computer is further configured to detect a raising of the upper eyelid based on the measurements of the reflections, and to increase a frame rate of the inward-facing head-mounted camera during the time the upper eyelid is raised above a threshold; whereby raising of the upper eyelid may indicate an emotional response comprising an expression of surprise.
7. The eye tracking system of claim 2 , wherein the computer is further configured to detect a raising of the lower eyelid based on the measurements of the reflections, and to increase a frame rate of the inward-facing head-mounted camera during the time the lower eyelid is raised above a threshold; whereby raising of the lower eyelid may indicate an emotional response comprising an expression of anger.
8. The eye tracking system of claim 1 , wherein in order to operate the inward-facing head-mounted camera according to the eyelid position the computer is configured to refrain from commanding the camera to capture the images during blinks and/or while the eye is closed.
9. The eye tracking system of claim 1 , wherein the computer is further configured to: extract eyelid positions by analyzing the images (image-based positions), and utilize the image-based positions and the measurements of the reflections to generate labeled training samples for training a machine learning-based model for detecting the eyelid position based on additional measurements of reflections taken with the head-mounted device.
10. The eye tracking system of claim 1 , wherein the computer is further configured to: receive photos from a camera belonging to a non-head-mounted device, extract eyelid positions by analyzing the photos (photo-based positions), and utilize the photo-based positions and the measurements of the reflections to generate labeled training samples for training a machine learning-based model for detecting the eyelid position based on additional measurements of reflections taken with the head-mounted device.
11. The eye tracking system of claim 10 , wherein the computer comprises a head-mounted computer and a non-head-mounted computer that are configured to communicate over a wireless communication channel; the non-head-mounted computer is configured to train the model, and the head-mounted computer is configured to use the trained model for detecting the eyelid position based on reflections.
12. The eye tracking system of claim 1 , wherein the head-mounted device comprises a photosensor-oculography device (PSOG) configured to measure reflections of light emitted by the PSOG towards the eye.
13. The eye tracking system of claim 1 , wherein the head-mounted device comprises an optical flow sensor configured to measure optical flow and/or visual motion, and the eyelid position is calculated based on an optical flow algorithm.
14. The eye tracking system of claim 1 , wherein the head-mounted device comprises range sensors configured to measure ranges between the range sensors and the eye; whereby measurements of the range sensors are affected by the eyelid position.
15. A method comprising:
capturing images of an eye by an inward-facing head-mounted camera;
emitting, by a head-mounted device, electromagnetic waves towards the eye, and taking measurements of reflections of the electromagnetic waves from the eye;
wherein average rate of taking measurements of the reflections is higher than average rate of capturing the images;
detecting, based on the measurements of the reflections, a position of at least one of the eyelids covering the eye (eyelid position); and
operating the inward-facing head-mounted camera according to the eyelid position in a manner that decreases, in the captured images, areas depicting the at least one of the eyelids.
16. The method of claim 15 , wherein the inward-facing head-mounted camera utilizes a sensor that supports changing of its region of interest (ROI), and further comprising utilizing the eyelid position to set the ROI around a portion of the eye that is not covered by the at least one of the eyelids.
17. The method of claim 15 , further comprising operating the inward-facing head-mounted camera according to the eyelid position in a manner that increases, in the captured images, areas depicting the iris; and wherein the average rate of taking measurements of the reflections is at least ten times the average rate of capturing the images.
18. The method of claim 15 , further comprising performing the following steps to train a machine learning-based model that is used for detecting the eyelid position based on additional measurements of reflections: extracting eyelid positions by analyzing the images (image-based positions); generating labels based on the image-based positions; generating feature values based on the measurements of the reflections; and providing the labels and feature values to train the machine learning-based model for detecting the eyelid position based on the additional measurements of the reflections.
19. The method of claim 15 , further comprising performing the following steps to train a machine learning-based model that is used for detecting the eyelid position based on additional measurements of reflections: receiving photos from a camera belonging to a non-head-mounted device, extracting eyelid positions by analyzing the photos (photo-based positions), generating labels based on the photo-based positions, generating feature values based on the measurements of the reflections, and providing the labels and feature values to train the machine learning-based model for detecting the eyelid position based on the additional measurements of the reflections.
20. A non-transitory computer readable medium storing one or more computer programs configured to cause a processor-based system to execute steps comprising:
capturing images of an eye by an inward-facing head-mounted camera;
emitting, by a head-mounted device, electromagnetic waves towards the eye, and taking measurements of reflections of the electromagnetic waves from the eye;
wherein average rate of taking measurements of the reflections is higher than average rate of capturing the images;
detecting, based on the measurements of the reflections, a position of at least one of the eyelids covering the eye (eyelid position); and
operating the inward-facing head-mounted camera according to the eyelid position in a manner that decreases, in the captured images, areas depicting the at least one of the eyelids.