Eye gaze tracking using neural networks

1. A method for characterizing a gaze position of a user in a query image, the method comprising:

obtaining a query image of a user captured by a camera of a mobile device;

obtaining device characteristics data specifying (i) characteristics of the mobile device, (ii) characteristics of the camera of the mobile device, or (iii) both;

maintaining data associating the device characteristics data with current values for a plurality of device-dependent parameters of a gaze prediction neural network; and

processing a neural network input comprising one or more images derived from the query image using the gaze prediction neural network, wherein:

the gaze prediction neural network has a plurality of parameters comprising (i) a plurality of device-independent parameters and (ii) the plurality of device-dependent parameters, wherein device-independent parameters are parameters that, after training, have the same values regardless of the device characteristics of the mobile device, and device-dependent parameters are parameters that have different values for different device characteristics of the mobile device;

the gaze prediction neural network is configured to, at run time and after the gaze prediction neural network has been trained, process the neural network input to generate a neural network output that characterizes a gaze position of the user in the query image; and

processing the neural network input comprises setting the values of the device-dependent parameters to the current values associated with the device characteristics data.

2. The method of claim 1 , wherein the obtaining the image, the obtaining the characteristics data, and the processing the neural network input are performed by the mobile device.

3. The method of claim 1 ,

wherein the gaze prediction neural network comprises a plurality of neural network layers configured to apply the device-independent-parameters to generate an initial neural network output that characterizes an initial predicted gaze position of the user, and

wherein the gaze prediction neural network is configured to adjust the initial neural network output in accordance with at least some of the device-dependent parameters to generate the neural network output.

4. The method of claim 3 , wherein adjusting the initial neural network output comprises:

applying a linear device dependent parameters transformation to the initial neural network output.

5. The method of claim 1 , wherein the neural network input further comprises data specifying a location of one or more eye landmarks in the query image.

6. The method of claim 5 , wherein the gaze prediction neural network is configured to:

apply at least some of the device-dependent parameters to adjust the location of the eye landmarks; and

process the adjusted location of the eye landmarks and the one or more images in accordance with the device-independent parameters to generate an initial neural network output.

7. The method of claim 1 , further comprising:

obtaining one or more calibration images of the user captured using the camera of the mobile device and, for each of the calibration images, a respective calibration label that labels a known gaze position of the user in the calibration image; and

training the gaze prediction neural network using the one or more calibration images to determine the current values for the set of device-dependent parameters from initial values for the set of device-dependent parameters while holding the device-independent parameters fixed.

8. The method of claim 1 , wherein the one or more images derived from the query image comprise a respective image crop corresponding to each of one or more eyes of the user.

9. A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations comprising:

obtaining a query image of a user captured by a camera of a mobile device;

obtaining device characteristics data specifying (i) characteristics of the mobile device, (ii) characteristics of the camera of the mobile device, or (iii) both;

maintaining data associating the device characteristics data with current values for a plurality of device-dependent parameters of a gaze prediction neural network; and

processing a neural network input comprising one or more images derived from the query image using the gaze prediction neural network, wherein:

the gaze prediction neural network has a plurality of parameters comprising (i) a plurality of device-independent parameters and (ii) the plurality of device-dependent parameters, wherein device-independent parameters are parameters that, after training, have the same values regardless of the device characteristics of the mobile device, and device-dependent parameters are parameters that have different values for different device characteristics of the mobile device;

the gaze prediction neural network is configured to, at run time and after the gaze prediction neural network has been trained, process the neural network input to generate a neural network output that characterizes a gaze position of the user in the query image; and

processing the neural network input comprises setting the values of the device-dependent parameters to the current values associated with the device characteristics data.

10. The system of claim 9 , wherein the obtaining the image, the obtaining the characteristics data, and the processing the neural network input are performed by the mobile device.

11. The system of claim 9 ,

wherein the gaze prediction neural network comprises a plurality of neural network layers configured to apply the device-independent-parameters to generate an initial neural network output that characterizes an initial predicted gaze position of the user, and

wherein the gaze prediction neural network is configured to adjust the initial neural network output in accordance with at least some of the device-dependent parameters to generate the neural network output.

12. The system of claim 11 , wherein adjusting the initial neural network output comprises:

applying a linear device dependent parameters transformation to the initial neural network output.

13. The system of claim 9 , wherein the neural network input further comprises data specifying a location of one or more eye landmarks in the query image.

14. The system of claim 13 , wherein the gaze prediction neural network is configured to:

apply at least some of the device-dependent parameters to adjust the location of the eye landmarks; and

process the adjusted location of the eye landmarks and the one or more images in accordance with the device-independent parameters to generate an initial neural network output.

15. The system of claim 9 , the operations further comprising:

obtaining one or more calibration images of the user captured using the camera of the mobile device and, for each of the calibration images, a respective calibration label that labels a known gaze position of the user in the calibration image; and

training the gaze prediction neural network using the one or more calibration images to determine the current values for the set of device-dependent parameters from initial values for the set of device-dependent parameters while holding the device-independent parameters fixed.

16. The system of claim 9 , wherein the one or more images derived from the query image comprise a respective image crop corresponding to each of one or more eyes of the user.

17. One or more non-transitory computer storage media encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising:

obtaining a query image of a user captured by a camera of a mobile device;

obtaining device characteristics data specifying (i) characteristics of the mobile device, (ii) characteristics of the camera of the mobile device, or (iii) both;

maintaining data associating the device characteristics data with current values for a plurality of device-dependent parameters of a gaze prediction neural network; and

processing a neural network input comprising one or more images derived from the query image using the gaze prediction neural network, wherein:

the gaze prediction neural network has a plurality of parameters comprising (i) a plurality of device-independent parameters and (ii) the plurality of device-dependent parameters, wherein device-independent parameters are parameters that, after training, have the same values regardless of the device characteristics of the mobile device, and device-dependent parameters are parameters that have different values for different device characteristics of the mobile device;

the gaze prediction neural network is configured to, at run time and after the gaze prediction neural network has been trained, process the neural network input to generate a neural network output that characterizes a gaze position of the user in the query image; and

processing the neural network input comprises setting the values of the device-dependent parameters to the current values associated with the device characteristics data.

18. The non-transitory computer storage media of claim 17 , wherein the obtaining the image, the obtaining the characteristics data, and the processing the neural network input are performed by the mobile device.

19. The non-transitory computer storage media of claim 17 ,

wherein the gaze prediction neural network comprises a plurality of neural network layers configured to apply the device-independent-parameters to generate an initial neural network output that characterizes an initial predicted gaze position of the user, and

wherein the gaze prediction neural network is configured to adjust the initial neural network output in accordance with at least some of the device-dependent parameters to generate the neural network output.

20. The non-transitory computer storage media of claim 17 , wherein the operations further comprise:

obtaining one or more calibration images of the user captured using the camera of the mobile device and, for each of the calibration images, a respective calibration label that labels a known gaze position of the user in the calibration image; and

training the gaze prediction neural network using the one or more calibration images to determine the current values for the set of device-dependent parameters from initial values for the set of device-dependent parameters while holding the device-independent parameters fixed.