Gaze detection pipeline in an artificial reality system
One embodiment of the present disclosure sets forth a method that includes receiving one or more two-dimensional images of one or more light patterns incident on an eye proximate to an eye region of a near-eye display device, and computing a gaze direction associated with the eye based on the one or more two-dimensional images.
1. A method, comprising:
receiving, from one or more sensor devices included in a near-eye display device, sensor data associated with an eye proximate to an eye region of the near-eye display device;
generating, based on the sensor data, a segmented three-dimensional depth map of the eye, wherein the segmented three-dimensional depth map identifies one or more components of the eye; and
applying at least one of a first trained machine learning model or a plane fitting technique to the segmented three-dimensional depth map to determine a gaze direction associated with the eye, wherein the gaze direction is based on a foveal angular offset with respect to a pupillary axis of the eye and an estimated center of the iris.
2. The method of claim 1 , wherein generating the segmented three-dimensional depth map comprises:
applying a second trained machine learning model to at least a portion of the sensor data to generate the segmented three-dimensional depth map, wherein the second trained machine learning model is trained on annotated sensor data associated with other eyes.
3. The method of claim 1 , wherein the first trained machine learning model is trained on annotated segmented three-dimensional depth maps of other eyes, and wherein one or more annotations on each of the annotated segmented three-dimensional depth maps identifies a corresponding gaze direction.
4. The method of claim 1 , wherein the plane fitting technique comprises:
determining a vector that is normal to a plane of an iris segment included in the segmented three-dimensional depth map, and
the gaze direction is associated with the vector.
5. The method of claim 1 , wherein the sensor data comprises at least one image of one or more light patterns incident on the eye.
6. The method of claim 1 , wherein the sensor data comprises one or more two-dimensional red, green, blue (RGB) images of the eye.
7. One or more non-transitory computer readable media storing instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of:
receiving, from one or more sensor devices included in a near-eye display device, sensor data associated with an eye proximate to an eye region of the near-eye display device;
generating, based on the sensor data, a segmented three-dimensional depth map of the eye, wherein the segmented three-dimensional depth map identifies one or more components of the eye; and
applying at least one of a first trained machine learning model or a plane fitting technique to the segmented three-dimensional depth map to determine a gaze direction associated with the eye, wherein the gaze direction is based on a foveal angular offset with respect to a pupillary axis of the eye and an estimated center of the iris.
8. The one or more non-transitory computer readable media of claim 7 , wherein generating the segmented three-dimensional depth map comprises:
applying a second trained machine learning model to at least a portion of the sensor data to generate the segmented three-dimensional depth map,
wherein the second trained machine learning model is trained on annotated sensor data associated with other eyes.
9. The one or more non-transitory computer readable media of claim 7 , wherein the first trained machine learning model is trained on annotated segmented three-dimensional depth maps of other eyes, and wherein one or more annotations on each of the annotated segmented three-dimensional depth maps identifies a corresponding gaze direction.
10. The one or more non-transitory computer readable media of claim 7 , wherein the plane fitting technique comprises determining a vector that is normal to a plane of an iris segment included in the segmented three-dimensional depth map, and wherein the gaze direction is associated with the vector.
11. A near-eye display system, comprising:
a structured light generator configured to project one or more light patterns into an eye region of a near-eye display device;
an image capture device configured to capture one or more two-dimensional images of the one or more light patterns incident on an eye proximate to the eye region; and
a feature generator configured to determine one or more features of the eye based on the captured one or more two-dimensional images by:
receiving, from the image capture device, the one or more two-dimensional images,
generating, based on the one or more two-dimensional images, a three-dimensional depth map of the eye,
segmenting the three-dimensional depth map to generate a segmented three-dimensional depth map of the eye, wherein the segmented three-dimensional depth map identifies one or more components of the eye, and
applying at least one of a trained machine learning model or a plane fitting technique to the segmented three-dimensional depth map to determine a gaze direction associated with the eye, wherein the gaze direction is based on a foveal angular offset with respect to a pupillary axis of the eye and an estimated center of the iris.
12. The near-eye display system of claim 11 , wherein the feature generator generates the three-dimensional depth map by extracting depth information associated with the eye from the one or more two-dimensional images.