Implicit calibration from screen content for gaze tracking
The technology relates to methods and systems for implicit calibration for gaze tracking. This can include receiving, by a neural network module, display content that is associated with presentation on a display screen ( 1202 ). The neural network module may also receive uncalibrated gaze information, in which the uncalibrated gaze information includes an uncalibrated gaze trajectory that is associated with a viewer gaze of the display content on the display screen ( 1204 ). A selected function is applied by the neural network module to the uncalibrated gaze information and the display content to generate a user-specific gaze function ( 1206 ). The user-specific gaze function has one or more personalized parameters. And the neural network module can then apply the user-specific gaze function to the uncalibrated gaze information to generate calibrated gaze information associated with the display content on the display screen ( 1208 ). Training and testing information may alternatively be created for implicit gaze calibration ( 1000 ).
1 . A computer-implemented method of performing implicit gaze calibration for gaze tracking, the method comprising:
receiving, by a neural network module, display content that is associated with presentation on a display screen;
receiving, by the neural network module, uncalibrated gaze information, the uncalibrated gaze information including an uncalibrated gaze trajectory that is associated with a viewer gaze of the display content on the display screen;
applying, by the neural network module, a selected function to the uncalibrated gaze information and the display content to generate a user-specific gaze function, the user-specific gaze function having one or more personalized parameters; and
applying, by the neural network module, the user-specific gaze function to the uncalibrated gaze information to generate calibrated gaze information associated with the display content on the display screen.
2 . The method of claim 1 , wherein the selected function is a linear or polynomial function.
3 . The method of claim 1 , wherein the uncalibrated gaze information further includes timestamp information for when the display content was collected.
4 . The method of claim 1 , wherein the uncalibrated gaze information further includes at least one of screen orientation information, camera focal length, aspect ratio, or resolution information.
5 . The method of claim 1 , wherein the one or more personalized parameters of the user-specific gaze function are estimated from collected data.
6 . The method of claim 1 , wherein applying the selected function to the uncalibrated gaze information and the display content to generate a user-specific gaze function includes:
generating temporal information and dimensional information at an encoder block of the neural network;
generating a context vector from the temporal information and the dimensional information in a self-attention block of the neural network; and
applying the context vector to the uncalibrated gaze information to generate the calibrated gaze information, in a decoder block of the neural network.
7 . The method of claim 6 , wherein the temporal information encompasses a selected time interval associated with a gaze along the display screen.
8 . The method of claim 7 , wherein the temporal information is encoded by looking through an entire sequence of gaze measurements and screen content pixels associated with the entire sequence.
9 . The method of claim 6 , wherein applying the context vector to the uncalibrated gaze information comprises multiplying the context vector with an array of data from the uncalibrated gaze information.
10 . The method of claim 6 , wherein applying the context vector to the uncalibrated gaze information includes applying the uncalibrated gaze information and the context vector using a plurality of fully connected layers of the neural network.
11 . The method of claim 1 , wherein the display content comprises synthetic content.
12 . The method of claim 11 , wherein the synthetic content includes at least one of synthetic text or synthetic graphical information.
13 . The method of claim 11 , wherein the synthetic content corresponds to a dataset of gaze trajectories for a group of users over a selected number of unique user interfaces.
14 . The system of claim 1 , wherein application of the selected function to the uncalibrated gaze information and the display content to generate a user-specific gaze function includes:
generation of temporal information and dimensional information at an encoder block of the neural network;
generation of a context vector from the temporal information and the dimensional information in a self-attention block of the neural network; and
application of the context vector to the uncalibrated gaze information to generate the calibrated gaze information, in a decoder block of the neural network.
15 . A system comprising one or more processors and one or more storage devices storing instructions, wherein when the instructions are executed by the one or more processors, the one or more processors implement a method of implicit gaze calibration for gaze tracking comprising:
receiving, by a neural network module of the one or more processors, display content that is associated with presentation on a display screen;
receiving, by the neural network module, uncalibrated gaze information, the uncalibrated gaze information including an uncalibrated gaze trajectory that is associated with a viewer gaze of the display content on the display screen;
applying, by the neural network module, a selected function to the uncalibrated gaze information and the display content to generate a user-specific gaze function, the user-specific gaze function having one or more personalized parameters; and
applying, by the neural network module, the user-specific gaze function to the uncalibrated gaze information to generate calibrated gaze information associated with the display content on the display screen.
16 . The system of claim 15 , wherein the uncalibrated gaze information further includes timestamp information for when the display content was collected.
17 . The system of claim 15 , wherein the uncalibrated gaze information further includes at least one of screen orientation information, camera focal length, aspect ratio, or resolution information.
18 . The system of claim 15 , wherein the one or more personalized parameters of the user-specific gaze function are estimated from collected data.
19 . The system of claim 15 , wherein the display content comprises synthetic content.
20 . The system of claim 15 , wherein the synthetic content either includes at least one of synthetic text or synthetic graphical information, or corresponds to a dataset of gaze trajectories for a group of users over a selected number of unique user interfaces.