On-the-fly calibration for improved on-device eye tracking
Calibration of eye tracking is improved by collecting additional calibration pairs when user is using apps with eye tracking. A user input component is presented on a display of an electronic device, detecting a dwelling action for user input component, and in response to detecting the dwelling action, obtaining a calibration pair comprising an uncalibrated gaze point and a screen location of the user input component, wherein the uncalibrated gaze point is determined based on an eye pose during the dwelling action. A screen gaze estimation is determine based on the uncalibrated gaze point, and in response to determining that the calibration pair is a valid calibration pair, training a calibration model using the calibration pair.
1 . A system comprising:
one or more processors; and
one or more non-transitory computer readable media comprising computer readable code executable by one or more processors to:
detect, by an electronic device, a dwelling action from a gaze of a user;
in response to detecting the dwelling action, determine an uncalibrated gaze point based on the gaze of the user;
determine a performance metric of a calibration model based on a comparison of an accuracy of the calibration model using ground truth calibration data and an accuracy of the calibration model incorporating the uncalibrated gaze point and a target gaze component; and
in response to determining that the performance metric satisfies a valid calibration criterion, incorporate, by the electronic device, gaze data for the gaze of the user into the calibration model, wherein the valid calibration criterion indicates that the performance of the calibration model improves with inclusion of the calibration pair.
2 . The system of claim 1 , wherein the uncalibrated gaze point is determined based on a left eye gaze target and a right eye gaze target.
3 . The system of claim 1 , wherein the calibration model is trained to utilize uncalibrated gaze data as input and output a gaze estimation.
4 . The system of claim 1 , wherein the computer readable code to determine the performance metric of the calibration model comprises computer readable code to:
generate a first prediction based on the ground truth calibration data and a first version of the calibration model prior to incorporating the uncalibrated gaze point and the target gaze component,
training a second version of the calibration model using the uncalibrated gaze point and the target gaze component, and
generate a second prediction based on the ground truth calibration data and the second version of the calibration model.
5 . The system of claim 1 , further comprising computer readable code to detect the dwelling action in response to detecting a calibration event, and wherein the computer readable code to detect the calibration event comprises computer readable code to:
determine a change in relative position of a user head and the electronic device; and
determine that the change in relative position of the user head and the electronic device satisfies a calibration event criterion.
6 . A method comprising:
detecting, by an electronic device, a dwelling action from a gaze of a user;
in response to detecting the dwelling action, determining an uncalibrated gaze point based on the gaze of the user;
determining a performance metric of a calibration model based on a comparison of an accuracy of the calibration model using ground truth calibration data and an accuracy of the calibration model incorporating the uncalibrated gaze point and a target gaze component; and
in response to determining that the performance metric satisfies a valid calibration criterion, incorporating, by the electronic device, gaze data for the gaze of the user into the calibration model, wherein the valid calibration criterion indicates that the performance of the calibration model improves with inclusion of the calibration pair.
7 . The method of claim 6 , wherein the uncalibrated gaze point is determined based on a left eye gaze target and a right eye gaze target.
8 . The method of claim 6 , wherein the calibration model is trained to utilize uncalibrated gaze data as input and output a gaze estimation.
9 . The method of claim 6 , wherein determining the performance metric of the calibration model comprises:
generating a first prediction based on the ground truth calibration data and a first version of the calibration model prior to incorporating the uncalibrated gaze point and the target gaze component, and
training a second version of the calibration model using the uncalibrated gaze point and the target gaze component,
generating a second prediction based on the ground truth calibration data and the second version of the uncalibrated gaze point and the target gaze component.
10 . The method of claim 6 , wherein the dwelling action is detected in response to a detected calibration event.
11 . The method of claim 6 , wherein the dwelling action from the gaze of the user is detected in response to a calibration event, wherein detecting the calibration event comprises:
determining a change in relative position of a user head and the electronic device; and
determining that the change in relative position of the user head and the electronic device satisfies a calibration event criterion.
12 . A non-transitory computer readable medium comprising computer readable code executable by one or more processors to:
detect, by an electronic device, a dwelling action from a gaze of a user;
in response to detecting the dwelling action, determine an uncalibrated gaze point based on the gaze of the user;
determine a performance metric of a calibration model based on a comparison of an accuracy of the calibration model using ground truth calibration data and an accuracy of the calibration model incorporating the uncalibrated gaze point and a target gaze component; and
in response to determining that the performance metric satisfies a valid calibration criterion, incorporate, by the electronic device, gaze data for the gaze of the user into the calibration model, wherein the valid calibration criterion indicates that the performance of the calibration model improves with inclusion of the calibration pair.
13 . The non-transitory computer readable medium of claim 12 , wherein the uncalibrated gaze point is determined based on a left eye gaze target and a right eye gaze target.
14 . The non-transitory computer readable medium of claim 12 , wherein the calibration model is trained to utilize uncalibrated gaze data as input and output a gaze estimation.
15 . The non-transitory computer readable medium of claim 12 , wherein the computer readable code to determine the performance metric of the calibration model comprises computer readable code to:
generate a first prediction based on the ground truth calibration data and a first version of the calibration model prior to incorporating the uncalibrated gaze point and the target gaze component,
training a second version of the calibration model using the uncalibrated gaze point and the target gaze component, and
generate a second prediction based on the ground truth calibration data and the second version of the calibration model.
16 . The non-transitory computer readable medium of claim 12 , wherein the dwelling action is detected in response to a detected calibration event.
17 . The non-transitory computer readable medium of claim 16 , further comprising computer readable code to:
determine a change in relative position of a user head and the electronic device; and
determine that the change in relative position of the user head and the electronic device satisfies a calibration event criterion.