Opengaze: gaze-tracking in the wild

1. A system comprising:

a camera positioned in an environment to capture image data of a subject;

a computing device communicatively coupled to the camera, the computing device comprising a processor and a non-transitory computer-readable memory; and

a machine-readable instruction set stored in the non-transitory computer-readable memory that causes the computing device to perform at least the following when executed by the processor:

receive the image data from the camera;

analyze the image data captured by the camera using a neural network trained on training data generated from a 360-degree panoramic camera configured to collect image data of a study subject and a visual target that is moved about the environment; and

predict a gaze direction vector of the subject with the neural network.

2. The system of claim 1 , wherein the machine-readable instruction set stored in the non-transitory computer-readable memory further causes the system to perform the following when executed by the processor:

predict, with the neural network, error bounds based on a pinball loss function.

3. The system of claim 2 , wherein the error bounds comprise at least one of a mean value, a 90% quantile value, or a 10% quantile value.

4. The system of claim 1 , wherein the predicted gaze direction vector is defined by a spherical coordinate system.

5. The system of claim 1 , wherein the predicted gaze direction vector comprises a three-dimensional gaze vector.

6. The system of claim 1 , wherein the training data comprises training image data of one or more subjects and corresponding training gaze direction vectors generated by:

collecting image data using a 360-degree panoramic camera of the one or more subjects where gazes of the one or more subjects are fixed on the visual target that is moved about the environment, and

calculating a corresponding training gaze direction vectors corresponding to the gazes of the one or more subjects.

7. The system of claim 1 , wherein image data is a head shot of the subject.

8. The system of claim 1 , wherein the neural network comprises one or more convolution layers and one or more fully connected layers.

9. The system of claim 1 , wherein the image data received by the processor comprises an image size of at least one of the following: 128×128, 227×227, and 384×384.

10. A method comprising:

receiving image data from a camera positioned in an environment to capture image data of a subject;

analyzing the image data with a neural network trained on training data generated from a 360-degree panoramic camera configured to collect image data of a study subject and a visual target that is moved about the environment; and

predicting a gaze direction vector of the subject with the neural network.

11. The method of claim 10 , further comprising:

predicting, with the neural network, error bounds based on a pinball loss function.

12. The method of claim 11 , wherein the error bounds comprise at least one of a mean value, a 90% quantile value, or a 10% quantile value.

13. The method of claim 10 , wherein the predicted gaze direction vector is defined by a spherical coordinate system.

14. The method of claim 10 , wherein the predicted gaze direction vector comprises a three-dimensional gaze vector.

15. The method of claim 10 , wherein the training data comprises training image data of one or more subjects and corresponding training gaze direction vectors generated by:

collecting image data using a 360-degree panoramic camera of the one or more subjects where gazes of the one or more subjects are fixed on the visual target that is moved about the environment, and

calculating a corresponding training gaze direction vectors corresponding to the gazes of the one or more subjects.

16. The method of claim 10 , image data is a head shot of the subject.

17. The method of claim 10 , wherein the neural network comprises one or more convolution layers and one or more fully connected layers.

18. The method of claim 10 , wherein the image data comprises an image size of at least one of the following: 128×128, 227×227, and 384×384.

19. A system comprising:

a camera positioned in an environment to capture image data of a subject;

a computing device communicatively coupled to the camera, the computing device comprising a processor and a non-transitory computer-readable memory;

a machine-readable instruction set stored in the non-transitory computer-readable memory that causes the computing device to perform at least the following when executed by the processor:

receive the image data from the camera;

analyze the image data captured by the camera using a neural network trained on training data comprising training image data of the one or more subjects and corresponding training gaze direction vectors generated by:

collecting image data using a 360-degree panoramic camera of one or more subjects where gazes of the one or more subjects are fixed on a visual target that is moved about the environment, and

calculating a corresponding training gaze direction vectors corresponding to the gazes of the one or more subjects;

predict, with the neural network, a gaze direction vector of the subject with the neural network; and

predict, with the neural network, error bounds based on a pinball loss function.

20. The system of claim 19 , wherein the error bounds comprise at least one of a mean value, a 90% quantile value, or a 10% quantile value.