Geometric calibration of head-worn multi-camera eye tracking system
View Patent ↗A system and methods for geometric calibration and use of head-worn multi-camera eye tracking system are provided. The system calibrates the internal and external parameters of the cameras as well as the parameters of a compact geometric model of the human eye. An exo-camera determines a relative pose with regard to a computer monitor and endo-cameras determine the eyeball position and orientation. An endo-exo calibration and an endo-eye calibration each with an optimization step are included. A homography and a joint optimization step are also provided. A single point calibration re-establishes calibration. The system in operational mode determines a user's gaze direction to perform a computer input function.
1. A method for calibrating a frame worn by a wearer containing an exo-camera directed to a display that is not worn by the wearer and a first and a second endo-camera, each directed to an eye of the wearer, comprising:
a processor collecting data related to a projection of an image with a plurality of calibration points on the display with a known set of 3D coordinates in coordinates of the exo-camera directed to the display and in coordinates of the endo-camera as an exo-endo pair;
the processor optimizing a relative pose of the exo-endo pair as a set of rotation and translation parameters which minimizes an error criterion;
the processor estimating parameters of a geometric model of the eye while a gaze by the eye in an unobstructed manner focuses on a known location on the display;
the processor minimizing a gaze projection error on the display with regard to a known ground truth; and
the processor determining a relative transformation between an endo-camera coordinate system and an exo-camera coordinate system; and wherein the relative transformation is expressed as:
p x =R ex p e t ex +
wherein
R ex εSO(3) is a rotation matrix, wherein SO(3) is a rotation group,
t ex εR 3 is a translation vector,
p x εR 3 is a vector of points in the exo camera coordinate system, and
p e εR 3 is a vector of points in the endo camera coordinate system.
2. The method of claim 1 , wherein a first exo-camera and endo-camera pair is formed for the first endo-camera and a second exo-camera and endo-camera pair is formed for the second endo-camera.
3. The method of claim 1 , wherein:
calibration grids G e , G x are placed around the endo-exo camera system such that G x is visible in the exo camera image and G e is visible in the endo camera image.
4. The method of claim 1 , wherein for each image establishing an exo-camera and endo-camera pair external pose matrices T n e εR 4 ×R 4 and T n e εR 4 ×R 4 are estimated from marked projected calibration points, wherein:
T n e is a endo-camera external pose matrix for image pair n, and
T n e is a exo-camera external pose matrix for image pair n.
5. The method of claim 4 , wherein the error criterion is determined by an optimization of an expression
σ
2
=
∑
n
=
1
N
[
{
Var
(
p
n
x
)
}
]
,
p
n
x
=
(
T
n
x
)
-
1
T
~
ex
T
n
e
p
e
,
wherein:
{tilde over (T)} ex is an estimation of T ex εR 4 ×R 4 which is a transformation matrix for homogeneous coordinates.
6. The method of claim 1 , wherein
the processor estimates the parameters of the of the geometric model of the eye by minimizing a projection error of an estimated intersection point of the direction of gaze with the display relative to an actual intersection point.
7. The method of claim 1 , further comprising:
minimizing a residual error between an estimated location on a display and an actual location on a display by estimating a homography.
8. The method of claim 6 , wherein the minimizing of the projection error generates estimated parameters that calibrate the frame relative to the display.
9. The method of claim 8 , wherein the estimated parameters are {tilde over (T)} ex , {tilde over (c)} and {tilde over (r)}, wherein
{tilde over (T)} ex is an estimation of a transformation matrix for homogeneous coordinates;
{tilde over (c)} is an estimate of a center of the eye; and
{tilde over (r)} is an estimate of a radius of the eye.
10. The method of claim 1 , further comprising:
the wearer after calibration looking at the screen and the processor determining from a post calibration exo-camera image and endo-camera image pair a coordinate on the display.
11. The method of claim 10 , wherein a gaze of the wearer is applied to provide a command to a computing device.
12. A method for calibrating a frame worn by a wearer containing an exo-camera directed to a display and a first and a second endo-camera, each directed to an eye of the wearer, comprising:
a processor collecting data related to a projection of an image on the display with a known set of 3D coordinates in coordinates of the exo-camera and in coordinates of the endo-camera as an exo-endo pair;
the processor optimizing a relative pose of the exo-endo pair as a set of rotation and translation parameters which minimizes an error criterion;
the processor estimating parameters of a geometric model of the eye while it focuses on a known location in the 3D screen space;
the processor minimizing a gaze projection error on the display with regard to a known ground truth, and wherein
the processor determines a relative transformation between an endo-camera coordinate system and an exo-camera coordinate system, the relative transformation is expressed as:
p x =R ex p e t ex+
wherein
R ex εSO(3) is a rotation matrix, wherein SO(3) is a rotation group,
t ex εR 3 is a translation vector,
p x εR 3 is a vector of points in the exo camera coordinate system, and
p e εR 3 is a vector of points in the endo camera coordinate system.
13. The method of claim 12 , wherein:
calibration grids G e , G x are placed around the endo-exo camera system such that G x is visible in the exo camera image and G e is visible in the endo camera image.
14. The method of claim 12 , wherein for each image establishing an exo-camera and endo-camera pair external pose matrices T n e εR 4 ×R 4 and T n x εR 4 ×R 4 are estimated from marked projected calibration points, wherein:
T n e is a endo-camera external pose matrix for image pair n, and
T n x is a exo-camera external pose matrix for image pair n.
15. The method of claim 14 , wherein the error criterion is determined by an optimization of an expression
σ
2
=
∑
n
=
1
N
[
{
Var
(
p
n
x
)
}
]
,
p
n
x
=
(
T
n
x
)
-
1
T
~
ex
T
n
e
p
e
,
wherein:
{tilde over (T)} ex is an estimation of T ex εR 4 ×R 4 which is a transformation matrix for homogeneous coordinates.
16. A method for calibrating a frame worn by a wearer containing an exo-camera directed to a display that is not worn by the wearer and a first and a second endo-camera, each directed to an eye of the wearer, comprising:
a processor collecting data related to a projection of an image with a plurality of calibration points on the display with a known set of 3D coordinates in coordinates of the exo-camera directed to the display and in coordinates of the endo-camera as an exo-endo pair;
the processor optimizing a relative pose of the exo-endo pair as a set of rotation and translation parameters which minimizes an error criterion;
the processor estimating parameters of a geometric model of the eye while it focuses on a known location on the display;
the processor minimizing a gaze projection error on the display with regard to a known ground truth; and
the processor minimizing a residual error between an estimated location on a display and an actual location on a display by estimating a homography, wherein the estimated homography is expressed as:
d=H{tilde over (d)},
wherein:
d is the actual location on the display,
{tilde over (d)} is the estimated location on the display, and
H is a matrix that determines a homography.
17. The method of claim 16 , wherein the minimizing of the projection error generates estimated parameters that calibrate the frame relative to the display and wherein the estimated parameters are {tilde over (T)} ex , {tilde over (c)} and {tilde over (r)}, wherein
{tilde over (T)} ex is an estimation of a transformation matrix for homogeneous coordinates;
{tilde over (c)} is an estimate of a center of the eye; and
{tilde over (r)} is an estimate of a radius of the eye.