Display systems and methods for determining registration between display and eyes of user
A display system may include a wearable display for rendering three-dimensional virtual image content that appears to be located in an environment of a user of the display. The relative positions of the display and one or more eyes of the user may not be in desired positions to receive, or register, image information outputted by the display. For example, the display-to-eye alignment may vary for different users and/or may change over time (e.g., as a user moves or as the display becomes displaced). The wearable device may determine a relative position and/or alignment between the display and the user's eyes by determining whether features of the eye are at certain vertical positions relative to the display. Based on the relative positions, the wearable device may determine if it is properly fitted to the user, and position render camera(s) accordingly to present virtual image content.
1 . A display system comprising:
a wearable display configured to project light into the eyes of a user to display virtual image content while the display is being worn by the user;
an eye-tracking camera system configured to image eyes of the user including a left eye and a right eye;
processing electronics comprising one or more processors; and
non-transitory computer-readable media storing instructions which, when executed by the processing electronics, cause the processing electronics to perform operations comprising:
based at least partly on one or more eye tracking images of the eyes that are captured by the eye-tracking camera system, calculating a left eye tracking confidence score indicative of a confidence level in a position of the left eye relative to the display and a right eye tracking confidence score indicative of a confidence level in a position of the right eye relative to the display;
based at least partly on the left eye tracking confidence score and the right eye tracking confidence score, determining at least one render camera position to position at least one render camera for rendering the virtual image content; and
causing the display to render the virtual image content based on the at least one render camera position,
wherein the left eye tracking confidence score is based on one or more of detecting a number of glints in the left eye, detecting that the user is blinking, detecting that the left eye is moving, and detecting that the pupil of the left eye is on a boundary of an eye tracking image, and
wherein the right eye tracking confidence score is based on one or more of detecting a number of glints in the right eye, detecting that the user is blinking, detecting that the right eye is moving, and detecting that the pupil of the right eye is on a boundary of an eye tracking image.
2 . The display system of claim 1 , wherein the wearable display comprises a stack of waveguides, wherein the waveguides are configured to output light to display the virtual image content, and wherein each waveguide comprises an incoupling optical element and an outcoupling optical element.
3 . The display system of claim 2 , wherein one of the waveguides outputs light with a different level of wavefront divergence than at least one other of the waveguides, and wherein different levels of wavefront divergence correspond to different depth planes.
4 . The display system of claim 1 , wherein the at least one render camera includes a first render camera for rendering the virtual image content to be viewed by the left eye and a second render camera for rendering the virtual image content to be viewed by the right eye.
5 . The display system of claim 1 , wherein the eye-tracking camera system comprises a first camera to image the left eye and a second camera to image the right eye.
6 . The display system of claim 1 , wherein the at least one render camera position includes a center of rotation of at least one of the eyes.
7 . The display system of claim 1 , wherein the at least one render camera position includes a center of perspective of at least one of the eyes.
8 . The display system of claim 1 , wherein the at least one render camera position includes a position along an axis between a center of rotation and a center of perspective of at least one of the eyes.
9 . The display system of claim 1 , wherein the operations further comprise:
based on at least one of the left eye tracking confidence score and the right eye tracking confidence score, determining a registration quality that indicates a degree to which the wearable display is properly registered to the eyes of the user, and wherein the at least one render camera position is based at least partly on the registration quality.
10 . The display system of claim 9 , wherein, when the left eye and the right eye are vertically offset from each other by less than a first predetermined threshold, determining the registration quality is based on a position of the left eye and a position of the right eye that is furthest from a desired vertical position.
11 . The display system of claim 10 , wherein, when the left eye and right eye are vertically offset from each other by less than a second predetermined threshold that is less than the first predetermined threshold, determining the registration quality is based on an average position of the left eye and an average position of the right eye.
12 . The display system of claim 10 , wherein, when the left eye and right eye are vertically offset from each other by more than the first predetermined threshold, determining the registration quality is based on the average position of the left eye and right eye.
13 . The display system of claim 9 , wherein, when one of the confidence scores is greater than the other, determining the registration quality is based on an eye of the user associated with the greater confidence score while an eye of the user associated with the lower confidence score is not used in determining the registration quality.
14 . A method for controlling a display system including a wearable display configured to project light into eyes of a user to display virtual image content while the display is being worn by the user, the method performed by processing electronics of the display system and comprising:
based at least partly on one or more eye tracking images of the eyes that are captured by an eye-tracking camera system of the display system, the eyes including a left eye and a right eye, calculating a left eye tracking confidence score indicative of a confidence level in a position of the left eye relative to the display and a right eye tracking confidence score indicative of a confidence level in a position of the right eye relative to the display;
based at least partly on the left eye tracking confidence score and the right eye tracking confidence score, determining at least one render camera position to position at least one render camera for rendering the virtual image content; and
causing the display to render the virtual image content based on the at least one render camera position,
wherein the left eye tracking confidence score is based on one or more of detecting a number of glints in the left eye, detecting that the user is blinking, detecting that the left eye is moving, and detecting that the pupil of the left eye is on a boundary of an eye tracking image, and
wherein the right eye tracking confidence score is based on one or more of detecting a number of glints in the right eye, detecting that the user is blinking, detecting that the right eye is moving, and detecting that the pupil of the right eye is on a boundary of an eye tracking image.
15 . The method of claim 14 , wherein the wearable display comprises a stack of waveguides, wherein the waveguides are configured to output light to display the virtual image content, and wherein each waveguide comprises an incoupling optical element and an outcoupling optical element.
16 . The method of claim 15 , wherein one of the waveguides outputs light with a different level of wavefront divergence than at least one other of the waveguides, and wherein different levels of wavefront divergence correspond to different depth planes.
17 . The method of claim 14 , wherein the at least one render camera position includes a center of rotation of at least one of the eyes.
18 . The method of claim 14 , wherein the at least one render camera position includes a center of perspective of at least one of the eyes.
19 . The method of claim 14 , wherein the at least one render camera position includes a position along an axis between a center of rotation and a center of perspective of at least one of the eyes.
20 . At least one non-transitory computer-readable storage media storing instructions which, when executed by processing electronics, cause the processing electronics to perform operations for controlling a display system including a wearable display configured to project light into eyes of a user to display virtual image content while the display is being worn by the user, the operations comprising:
based at least partly on one or more eye tracking images of the eyes that are captured by an eye-tracking camera system of the display system, the eyes including a left eye and a right eye, calculating a left eye tracking confidence score indicative of a confidence level in a position of the left eye relative to the display and a right eye tracking confidence score indicative of a confidence level in a position of the right eye relative to the display;
based at least partly on the left eye tracking confidence score and the right eye tracking confidence score, determining at least one render camera position to position at least one render camera for rendering the virtual image content; and
causing the display to render the virtual image content based on the at least one render camera position,
wherein the left eye tracking confidence score is based on one or more of detecting a number of glints in the left eye, detecting that the user is blinking, detecting that the left eye is moving, and detecting that the pupil of the left eye is on a boundary of an eye tracking image, and
wherein the right eye tracking confidence score is based on one or more of detecting a number of glints in the right eye, detecting that the user is blinking, detecting that the right eye is moving, and detecting that the pupil of the right eye is on a boundary of an eye tracking image.