Eye gaze tracking using neural networks
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for characterizing a gaze position of a user in a query image. One of the methods includes obtaining a query image of a user captured by a camera of a mobile device; obtaining device characteristics data specifying (ii) characteristics of the mobile device, (ii) characteristics of the camera of the mobile device, or (iii) both; and processing a neural network input comprising (i) one or more images derived from the query image and (ii) the device characteristics data using a gaze prediction neural network, wherein the gaze prediction neural network is configured to, at run time and after the gaze prediction neural network has been trained, process the neural network input to generate a neural network output that characterizes a gaze position of the user in the query image.
1. A method performed by one or more computers, the method comprising:
obtaining, by a mobile device, a query image of a user captured by a camera of the mobile device;
obtaining, by the mobile device, device characteristics data characterizing one or more of a manufacturer of the mobile device, a model of the mobile device, a current orientation of the camera, or an indication of whether certain settings of the camera or enabled or disabled;
maintaining, by the mobile device, data associating the device characteristics data with current values for a plurality of device-dependent parameters of a gaze prediction neural network, wherein the current values of the plurality of device-dependent parameters have been determined by training the gaze prediction neural network on calibration examples each including the device characteristics data;
processing, by the mobile device, a neural network input comprising one or more images derived from the query image using the gaze prediction neural network, wherein:
the gaze prediction neural network has a plurality of parameters comprising i) a plurality of device-independent parameters that have been determined by training the gaze prediction neural network on training examples each including respective device characteristics data and ii) the plurality of device-dependent parameters, wherein device-independent parameters are parameters that, after training, have the same values regardless of the device characteristics data, and device-dependent parameters are parameters that have different values for different device characteristics of the mobile device;
the gaze prediction neural network is configured to, at run time and after the gaze prediction neural network has been trained, process the neural network input to generate a neural network output that characterizes a gaze position of the user in the query image; and
processing the neural network input comprises setting the values of the device-dependent parameters to the current values associated with the device characteristics data; and
controlling, by the mobile device, one or more functions of the mobile device based on the neural network output.
2. The method of claim 1 , wherein controlling one or more functions of the mobile device comprises providing an input to an operating system of the mobile device.
3. The method of claim 1 ,
wherein the gaze prediction neural network comprises a plurality of neural network layers configured to apply the device independent-parameters to generate an initial neural network output that characterizes an initial predicted gaze position of the user, and
wherein the gaze prediction neural network is configured to adjust the initial neural network output in accordance with at least some of the device-specific parameters to generate the neural network output.
4. The method of claim 3 , wherein adjusting the initial neural network output comprises:
applying a linear device dependent parameters transformation to the initial neural network output.
5. The method of claim 1 , wherein the neural network input further comprises data specifying a location of one or more eye landmarks in the query image.
6. The method of claim 5 , wherein the gaze prediction neural network is configured to:
apply at least some of the device-specific parameters to adjust the location of the eye landmarks; and
process the adjusted location of the eye landmarks and the one or more images in accordance with the device-independent parameters to generate an initial neural network output.
7. The method of claim 1 , further comprising:
obtaining one or more calibration images of the user captured using the camera of the mobile device and, for each of the calibration images, a respective calibration label that labels a known gaze position of the user in the calibration image; and
training the gaze prediction neural network using the one or more calibration images to determine the current values for the set of device-dependent parameters from initial values for the set of device-dependent parameters while holding the device-independent parameters fixed.
8. The method of claim 1 , wherein the one or more images derived from the query image comprise a respective image crop corresponding to each of one or more eyes of the user.
9. A method comprising:
obtaining, by a mobile device, a query image of a user captured by a camera of the mobile device;
obtaining, by the mobile device, device characteristics data characterizing one or more of a manufacturer of the mobile device, a model of the mobile device, a current orientation of the camera, or an indication of whether certain settings of the camera or enabled or disabled;
processing, by the mobile device, a neural network input comprising (i) one or more images derived from the query image, (ii) the device characteristics data, and (iii) data specifying respective locations of one or more eye landmarks in the query image using a gaze prediction neural network to generate a neural network output that characterizes a gaze position of the user in the query image, the processing comprising:
adjusting the locations of the eye landmarks in the query image according to the device characteristics data, and
processing the adjusted locations of the eye landmarks in the query image and the one or more images to generate the neural network output that characterizes the gaze position of the user in the query image; and
controlling, by the mobile device, one or more functions of the mobile device based on the neural network output.
10. The method of claim 9 , wherein controlling one or more functions of the mobile device comprises providing an input to an operating system of the mobile device.
11. The method of claim 9 , wherein the gaze prediction neural network has a plurality of device-independent parameters and a plurality of device-dependent parameters, the method further comprising:
maintaining data associating the device characteristics data with current values for the plurality of device-dependent parameters,
wherein processing the neural network input comprises setting the values of the device-dependent parameters to the current values associated with the device characteristics data.
12. The method of claim 9 , wherein processing the adjusted locations of the eye landmarks and the one or more images to generate the neural network output comprises:
processing the adjusted locations of the eye landmarks and the one or more images in accordance with the device characteristics data to generate the neural network output.
13. The method of claim 9 , further comprising:
obtaining one or more calibration images of the user captured using the camera of the mobile device and, for each of the calibration images, a respective calibration label that labels a known gaze position of the user in the calibration image; and
training the gaze prediction neural network using the one or more calibration images.
14. The method of claim 9 , wherein the one or more images derived from the query image comprise a respective image crop corresponding to each of one or more eyes of the user.
15. The method of claim 9 , wherein the neural network output comprises a measure of central tendency and a measure of variance of a distribution over possible gaze positions of the user in the query image.
16. A mobile device comprising one or more storage devices storing instructions that when executed by the mobile cause the mobile device to perform operations comprising:
obtaining, by a mobile device, a query image of a user captured by a camera of the mobile device;
obtaining, by the mobile device, device characteristics data characterizing one or more of a manufacturer of the mobile device, a model of the mobile device, a current orientation of the camera, or an indication of whether certain settings of the camera or enabled or disabled;
maintaining, by the mobile device, data associating the device characteristics data with current values for a plurality of device-dependent parameters of a gaze prediction neural network, wherein the current values of the plurality of device-dependent parameters have been determined by training the gaze prediction neural network on calibration examples each including the device characteristics data;
processing, by the mobile device, a neural network input comprising one or more images derived from the query image using the gaze prediction neural network, wherein:
the gaze prediction neural network has a plurality of parameters comprising i) a plurality of device-independent parameters that have been determined by training the gaze prediction neural network on training examples each including respective device characteristics data and ii) the plurality of device-dependent parameters, wherein device-independent parameters are parameters that, after training, have the same values regardless of the device characteristics data, and device-dependent parameters are parameters that have different values for different device characteristics of the mobile device;
the gaze prediction neural network is configured to, at run time and after the gaze prediction neural network has been trained, process the neural network input to generate a neural network output that characterizes a gaze position of the user in the query image; and
processing the neural network input comprises setting the values of the device-dependent parameters to the current values associated with the device characteristics data; and
controlling, by the mobile device, one or more functions of the mobile device based on the neural network output.
17. The system of claim 16 , wherein controlling one or more functions of the mobile device comprises providing an input to an operating system of the mobile device.
18. The system of claim 16 ,
wherein the gaze prediction neural network comprises a plurality of neural network layers configured to apply the device independent-parameters to generate an initial neural network output that characterizes an initial predicted gaze position of the user, and
wherein the gaze prediction neural network is configured to adjust the initial neural network output in accordance with at least some of the device-specific parameters to generate the neural network output.
19. The system of claim 18 , wherein adjusting the initial neural network output comprises:
applying a linear device dependent parameters transformation to the initial neural network output.
20. The system of claim 16 , wherein the neural network input further comprises data specifying a location of one or more eye landmarks in the query image.