DYNAMIC CALIBRATION SYSTEMS AND METHODS FOR WEARABLE HEADS-UP DISPLAYS
Systems, methods and articles that provide dynamic calibration of eye tracking systems for wearable heads-up displays (WHUDs). The eye tracking system may determine a user's gaze location on a display of the WHUD utilizing a calibration point model that includes a plurality of calibration points. During regular use of the WHUD by the user, the calibration point model may be dynamically updated based on the user's interaction with user interface (UI) elements presented on the display. The UI elements may be specifically designed (e.g., shaped, positioned, displaced) to provide in-use and on-going dynamic calibration of the eye tracking system, which in at least some implementations may be unnoticeable to the user.
1 . A wearable heads-up display (WHUD), comprising:
a support frame;
a display carried by the support frame;
a glint detection module carried by the support frame that, in operation, determines glint space points in a glint space that correspond to a region in a field of view of an eye of a user at which a gaze of the eye is directed;
at least one processor carried by the support frame, the at least one processor communicatively coupled to the display and the glint detection module; and
at least one nontransitory processor-readable storage medium carried by the support frame, the at least one nontransitory processor-readable storage medium communicatively coupled to the at least one processor, wherein the at least one nontransitory processor-readable storage medium stores data or processor-executable instructions that, when executed by the at least one processor, cause the at least one processor to:
obtain one or more calibration point models each comprising a plurality of calibration points, each calibration point comprising:
a glint space point in a glint space captured by the glint detection module, the glint space point representative of a position of an eye of a user of the WHUD; and
a display space point in a display space of the display, the display space point representative of a location on the display at which a gaze of the user is inferred to be resting when the glint space point is captured by the glint detection module;
generate a transform from the glint space to the display space for each of the calibration point models;
determine user gaze location in the display space using received glint information and the generated transform;
from time-to-time during regular operation of the WHUD by the user,
generate at least one additional calibration point;
add the additional calibration point to at least one of the calibration point models to generate one or more child calibration point models;
generate a transform for each of the one or more child calibration point models; and
determine a user gaze location in the display space using at least one glint space point received from the glint detection module and at least one transform of the one or more child calibration point models.
2 . The WHUD of claim 1 wherein the at least one processor generates an additional inferred calibration point comprising:
a glint space point received from the glint detection module; and
a display space point that corresponds to a location in the display space of a UI element determined to be the user gaze location.
3 . The WHUD of claim 1 wherein the at least one processor generates an additional inferred calibration point for each of a plurality of UI elements displayed on the display, each inferred calibration point comprising:
a glint space point received from the glint detection module; and
a display space point that corresponds to a location in the display space of one of a plurality of UI elements.
4 . The WHUD of claim 1 wherein the at least one processor generates an additional selected calibration point comprising:
a glint space point received from the glint detection module; and
a display space point that is a location of a UI element on the display selected by the user during regular operation of the WHUD.
5 . The WHUD of claim 4 wherein the at least one processor:
determines a user gaze location in the display space using at least one glint space point received from the glint detection module and the one or more child calibration point models that include the additional selected calibration point.
6 . The WHUD of claim 1 wherein the at least one processor:
generates an affine transform from the glint space to the display space.
7 . The WHUD of claim 1 wherein the at least one processor:
solves a matrix utilizing at least one of a QR decomposition method or singular value decomposition method.
8 . The WHUD of claim 1 wherein the at least one processor:
from time-to-time during regular operation of the WHUD by the user, evicts at least one calibration point from a calibration point model.
9 . The WHUD of claim 8 wherein the at least one processor:
evicts an oldest calibration point from the calibration point model.
10 . The WHUD of claim 8 wherein the at least one processor:
evicts a calibration point based on at least one of the locations of calibration points in the calibration point model or the times at which the calibration points in the calibration point model were obtained.
11 . The WHUD of claim 1 wherein, to obtain a calibration point model comprising a plurality of calibration points, the at least one processor:
populates the display of the WHUD with a plurality of UI elements; and
for each of the plurality of UI elements,
receives a selection of the UI element by the user;
receives a glint space point from the glint detection module obtained concurrently with the selection of the UI element by the user; and
generates a calibration point that comprises the received glint space point and a display space point representative of the location of the UI element on the display of the WHUD.
12 . The WHUD of claim 11 wherein the at least one processor:
populates the display of the WHUD with a plurality of UI elements one at a time in a sequential order.
13 . The WHUD of claim 1 wherein, to obtain a calibration point model comprising a plurality of calibration points, the at least one processor:
causes four UI elements to be sequentially displayed on the display, each of the four UI elements sequentially displayed in a different one of four corners of the display; and
obtains four calibration points that each correspond to a respective one of the UI elements, each calibration point comprising a display point the display space and a glint space point in the glint space.
14 . The WHUD of claim 1 wherein, to obtain a calibration point model comprising a plurality of calibration points, the at least one processor:
causes a UI element to move on the display of the WHUD according to a determined pattern; and
generates a plurality of calibration points as the UI element moves on the display, each calibration point comprises:
a glint space point in the glint space captured by the glint detection module; and
a display space point in the display space, the display space point representative of a location on the display of the moving UI element when the corresponding glint space point is captured by the glint detection module.
15 . The WHUD of claim 14 wherein the determined pattern comprises a rectangular-shaped pattern, and the at least one processor:
causes the UI element to move in the rectangular-shaped pattern in a first direction; and
causes the UI element to move in the rectangular-shaped pattern in a second direction, the second direction opposite the first direction.
16 . The WHUD of claim 1 wherein the at least one processor:
receives at least one auxiliary sensor value from at least one auxiliary sensor during regular operation of the WHUD by the user; and
optimizes a transform of at least one calibration point model based at least in part on the received at least one auxiliary sensor value.
17 . The WHUD of claim 16 wherein the at least one auxiliary sensor comprises at least one of a proximity sensor, a gyroscope sensor or an accelerometer.
18 . The WHUD of claim 1 wherein the at least one processor:
receives a plurality of calibration points, each calibration point comprising:
a glint space point;
a display space point; and
at least one auxiliary sensor value from at least one auxiliary sensor obtained concurrently with the glint space point and the display space point; and
trains a machine learning model utilizing the plurality of calibration points, or data derived therefrom, the trained machine learning model receives as inputs at least one current auxiliary sensor value and outputs at least one of a set of calibration points or transform parameters.
19 . The WHUD of claim 18 wherein the at least one processor:
optimizes at least one transform utilizing the trained machine learning model.
20 . The WHUD of claim 18 wherein the at least one processor:
receives a plurality of calibration points from the WHUD and from a population of WHUDs operated by a population of users.