System and method for continuous calibration of eye tracking systems used in VR/AR HMD units
A virtual reality/augmented reality (VR/AR) wearable assembly is described herein. The VR/AR wearable assembly includes a wearable frame adapted to be worn over a patient's eyes, a pair of photosensor oculography (PSOG) assemblies mounted to the wearable frame with each PSOG assembly including a display and an eye tracking assembly, and a processor coupled to each PSOG assembly. The processor is configured to execute an algorithm to continuously calibrate the eye tracking assembly including the steps of rendering a sequence of images on a corresponding display and calibrating a corresponding eye tracking assembly by establishing a display coordinate system associated with the corresponding display, determining predicted fixation locations and estimated gaze position locations within the display coordinate system, and determining an offset gaze position value by mapping the estimated gaze position location to the predicted fixation location within the display coordinate system.
1. A virtual reality/augmented reality (VR/AR) wearable assembly comprising:
a wearable frame adapted to be worn over a user's eyes;
a display system mounted to the wearable frame and configured to project digital image onto the user's field of vision;
a pair of photosensor oculography (PSOG) eye tracking assemblies mounted to the wearable frame such that each PSOG eye tracking assembly is positioned adjacent a corresponding eye of the user, each PSOG eye tracking assembly including a plurality of groups of photosensors and light emitters; and
one or more processors coupled to each PSOG eye tracking assembly and the display system, the one or more processors configured to execute an algorithm to calibrate the each PSOG eye tracking assembly including the steps of:
rendering a sequence of images on the display system; and
calibrating a corresponding PSOG eye tracking assembly by:
determining a predicted fixation location associated with a corresponding rendered image;
determining an estimated gaze position location of the corresponding eye of the user when viewing the corresponding rendered image by alternating illumination of the groups of light emitters to generate predefined lighting patterns;
determining an offset gaze position value by mapping the estimated gaze position location to the predicted fixation location; and
determining a corrected gaze position location of the corresponding eye of the patient user based on the estimated gaze position location and the determined offset gaze position value.
2. The VR/AR wearable assembly of claim 1 , further comprising a headset shift sensor configured to monitor a relative position between the wearable frame and a head of the user, the one or more processors is configured to execute the algorithm including the steps of calibrating each PSOG eye tracking assembly upon detecting a shift in the relative position between the wearable frame and the head of the user.
3. The VR/AR wearable assembly of claim 1 , wherein the one or more processors is configured to execute the algorithm including the steps of:
rendering the sequence of images on the display system using foveated rendering.
4. The VR/AR wearable assembly of claim 1 , wherein the one or more processors is configured to execute the algorithm including the steps of:
rendering the sequence of images on the display system including active image objects selectable by detected gestures associated with user eye movements.
5. The VR/AR wearable assembly of claim 1 , wherein the one or more processors is configured to execute the algorithm including the steps of:
estimating inter-pupilar distance (IPD) of the user.
6. The VR/AR wearable assembly of claim 1 , wherein the one or more processors is configured to execute the algorithm including the steps of:
establishing a display coordinate system associated with the corresponding display;
determining the predicted fixation location within the display coordinate system associated with the corresponding rendered image;
determining the estimated gaze position location within the display coordinate system; and
determining the offset gaze position value by mapping the estimated gaze position location to the predicted fixation location within the display coordinate system.
7. The VR/AR wearable assembly of claim 6 , wherein the processor is configured to execute the algorithm including the steps of:
generating a saliency map associated with the corresponding rendered image including display coordinates of the predicted fixation location within the display coordinate system;
mapping a location of the estimated gaze position location onto the saliency map; and
determining the offset gaze position value based on the mapped location of the estimated gaze position on the saliency map.
8. A method of operating a virtual reality/augmented reality (VR/AR) wearable assembly including a wearable frame adapted to be worn over a user's eyes, a display system mounted to the wearable frame and configured to project digital image onto the user's field of vision, a pair of photosensor oculography (PSOG) eye tracking assemblies mounted to the wearable frame with each PSOG eye tracking assembly positioned adjacent a corresponding eye of the user and including a plurality of groups of photosensors and light emitters, and one or more processors coupled to each PSOG eye tracking assembly and the display system, the method including the one or more processors performing an algorithm to calibrate each PSOG eye tracking assembly including the steps of:
rendering a sequence of images on the display system; and
calibrating a corresponding PSOG eye tracking assembly by:
determining a predicted fixation location associated with a corresponding rendered image;
determining an estimated gaze position location of the corresponding eye of the user when viewing the corresponding rendered image by alternating illumination of the groups of light emitters to generate predefined lighting patterns;
determining an offset gaze position value by mapping the estimated gaze position location to the predicted fixation location; and
determining a corrected gaze position location of the corresponding eye of the user based on the estimated gaze position location and the determined offset gaze position value.
9. The method of claim 8 , including the one or more processors performing the algorithm including the steps of:
calibrating each PSOG eye tracking assembly upon detecting a shift in a relative position between the wearable frame and a head of the user using a headset shift sensor configured to monitor the relative position between the wearable frame and the head of the user.
10. The method of claim 8 , including the one or more processors performing the algorithm including the steps of:
rendering the sequence of images on the display system using foveated rendering.
11. The method of claim 8 , including the one or more processors performing the algorithm including the steps of:
rendering the sequence of images on the display system including active image objects selectable by detected gestures associated with user eye movements.
12. The method of claim 8 , including the one or more processors performing the algorithm including the steps of:
estimating inter-pupilar distance (IPD) of the user.
13. The method of claim 8 , including the one or more processors performing the algorithm including the steps of:
establishing a display coordinate system associated with the corresponding display;
determining the predicted fixation location within the display coordinate system associated with the corresponding rendered image;
determining the estimated gaze position location within the display coordinate system; and
determining the offset gaze position value by mapping the estimated gaze position location to the predicted fixation location within the display coordinate system.
14. The method of claim 13 , including the one or more processors performing the algorithm including the steps of:
generating a saliency map associated with the corresponding rendered image including display coordinates of the predicted fixation location within the display coordinate system;
mapping a location of the estimated gaze position location onto the saliency map; and
determining the offset gaze position value based on the mapped location of the estimated gaze position on the saliency map.
15. A non-transitory computer-readable storage media having computer-executable instructions embodied thereon to operate a virtual reality/augmented reality (VR/AR) wearable assembly including a wearable frame adapted to be worn over a user's eyes, a display system mounted to the wearable frame and configured to project digital image onto the user's field of vision, a pair of photosensor oculography (PSOG) eye tracking assemblies mounted to the wearable frame with each PSOG eye tracking assembly positioned adjacent a corresponding eye of the user and including a plurality of groups of photosensors and light emitters, and one or more processors coupled to each PSOG eye tracking assembly and the display system, when executed by the one or more processors the computer-executable instructions cause the one or more processors to perform an algorithm to calibrate each PSOG eye tracking assembly including the steps of:
rendering a sequence of images on the display system; and
calibrating a corresponding PSOG eye tracking assembly by:
determining a predicted fixation location associated with a corresponding rendered image;
determining an estimated gaze position location of the corresponding eye of the user when viewing the corresponding rendered image by alternating illumination of the groups of light emitters to generate predefined lighting patterns;
determining an offset gaze position value by mapping the estimated gaze position location to the predicted fixation location; and
determining a corrected gaze position location of the corresponding eye of the user based on the estimated gaze position location and the determined offset gaze position value.
16. The non-transitory computer-readable storage media of claim 15 , wherein the computer-executable instructions cause the one or more processors to perform the algorithm including the steps of:
calibrating each PSOG eye tracking assembly upon detecting a shift in a relative position between the wearable frame and a head of the user using a headset shift sensor configured to monitor the relative position between the wearable frame and the head of the user.
17. The non-transitory computer-readable storage media of claim 15 , wherein the computer-executable instructions cause the one or more processors to perform the algorithm including the steps of:
rendering the sequence of images on the display system using foveated rendering.
18. The non-transitory computer-readable storage media of claim 15 , wherein the computer-executable instructions cause the one or more processors to perform the algorithm including the steps of:
rendering the sequence of images on the display system including active image objects selectable by detected gestures associated with user eye movements.
19. The non-transitory computer-readable storage media of claim 15 , wherein the computer-executable instructions cause the one or more processors to perform the algorithm including the steps of:
estimating inter-pupilar distance (IPD) of the user.
20. The non-transitory computer-readable storage media of claim 15 , wherein the computer-executable instructions cause the one or more processors to perform the algorithm including the steps of:
establishing a display coordinate system associated with the corresponding display;
determining the predicted fixation location within the display coordinate system associated with the corresponding rendered image;
determining the estimated gaze position location within the display coordinate system; and
determining the offset gaze position value by mapping the estimated gaze position location to the predicted fixation location within the display coordinate system.