Enhanced eye tracking techniques based on neural network analysis of images
Enhanced eye-tracking techniques for augmented or virtual reality display systems. An example method includes obtaining an image of an eye of a user of a wearable system, the image depicting glints on the eye caused by respective light emitters, wherein the image is a low dynamic range (LDR) image; generating a high dynamic range (HDR) image via computation of a forward pass of a machine learning model using the image; determining location information associated with the glints as depicted in the HDR image, wherein the location information is usable to inform an eye pose of the eye.
1. A computer-implemented system, comprising:
one or more computers of an augmented reality or virtual reality wearable system; and
one or more computer memory devices interoperably coupled with the one or more computers and having tangible, non-transitory, machine-readable media storing one or more instructions that, when executed by the one or more computers, perform one or more operations, comprising:
setting one or more parameters associated with an eyeball camera rig, the eyeball camera rig comprising a representation of an eyeball, a plurality of light emitters, and one or more imaging devices, and the one or more parameters indicating, at least, an orientation associated with the representation of an eyeball;
obtaining, as an obtained plurality of images, a plurality of images of the representation of an eyeball, wherein the obtained plurality of images are low dynamic range (LDR) images obtained at different exposures;
generating a high dynamic range (HDR) image based on the obtained plurality of images; and
causing a machine learning model to be trained based on at least one image of the obtained plurality of images and the HDR image, wherein the machine learning model is trained to generate an HDR image from an input LDR image.
2. The computer-implemented system of claim 1 , wherein the representation of an eyeball comprises one or more lenses and other optical components.
3. The computer-implemented system of claim 1 , wherein the one or more parameters comprise one or more of pulse width modulation of the light emitters or exposure information for the imaging devices.
4. The computer-implemented system of claim 1 , wherein the one or more parameters comprise background light, position of background light, or color or wavelength spectrum information of one or more background lights.
5. The computer-implemented system of claim 1 , wherein for each orientation of the representation of an eyeball, simulating different real-world environments by adjusting one or more background lights.
6. The computer-implemented system of claim 1 , wherein the obtained plurality of images are obtained with a particular exposure associated with an imaging device of the one or more imaging devices.
7. The computer-implemented system of claim 1 , wherein the light emitters are light-emitting diodes or infrared light-emitting diodes.
8. The computer-implemented system of claim 1 , wherein the machine learning model is a convolutional neural network.
9. The computer-implemented system of claim 1 , wherein the machine learning model is an autoencoder with a plurality of convolutional layers, the autoencoder including an encoder portion which generates a latent feature representation associated with the at least one image of the obtained plurality of images and a decoder portion which generates the HDR image based on the latent feature representation.
10. The computer-implemented system of claim 9 , wherein one or more skip connections connect the encoder portion and the decoder portion, wherein the one or more skip connections provide domain transfer information from the encoder portion to the decoder portion.
11. The computer-implemented system of claim 10 , wherein the domain transfer information comprises logarithmic HDR values generated from LDR display values of the image.
12. The computer-implemented system of claim 1 , wherein the machine learning model was trained to generate, based on training data, an HDR image from a corresponding LDR image, wherein the training data comprises a multitude of LDR image and HDR image pairs, and wherein the multitude of LDR image and HDR image pairs are generated using the eyeball camera rig.
13. The computer-implemented system of claim 1 , wherein the HDR image is generated by combining the obtained plurality of images.
14. The computer-implemented system of claim 1 , wherein the obtained plurality of images depict glints on the representation of an eyeball caused by respective light emitters.
15. The computer-implemented system of claim 14 , comprising determining location information associated with the glints on the representation of an eyeball as depicted in the HDR image.
16. The computer-implemented system of claim 15 , wherein the location information is usable to inform an eye pose of the representation of an eyeball.
17. The computer-implemented system of claim 14 , wherein the glints are depicted on a pupil of the representation of an eyeball.
18. The computer-implemented system of claim 14 , wherein the glints depicted in the obtained plurality of images are of a greater size than glints depicted in the HDR image.
19. The computer-implemented system of claim 14 , wherein each glint depicted in the obtained plurality of images includes a larger irregular portion than a corresponding glint included in the HDR image.
20. The computer-implemented system of claim 19 , wherein the larger irregular portion is a tail extending from an oval or circular portion of the glint or the larger irregular portion is a tail extending from an oval or circular portion of the glint.