Handling multiple eye-gaze tracking (EGT) sources in virtual, augmented, or mixed reality (XR) applications
Embodiments of systems and methods for handling multiple Eye-Gaze Tracking (EGT) sources in virtual, augmented, or mixed reality (xR) applications are described. In some embodiments, an Information Handling System (IHS) may include a processor and a memory coupled to the processor, the memory having program instructions stored thereon that, upon execution by the processor, cause the IHS to: receive a first EGT vector from a first EGT system and a second EGT vector from a second EGT system; calculate a composite EGT vector using a first weight associated with the first EGT system and a second weight associated with the second EGT system; and render an image, in part, based upon the composite EGT vector.
1. An Information Handling System (IHS) for handling multiple Eye-Gaze Tracking (EGT) sources in virtual, augmented, or mixed reality applications, comprising:
a processor; and
a memory coupled to the processor, the memory having program instructions stored thereon that, upon execution by the processor, cause the IHS to:
receive a first Eye-Gaze Tracking (EGT) vector from a first EGT system coupled to a Head-Mounted Display (HMD);
receive a second EGT vector from a second EGT system coupled to an external display of a host Information Handling System (IHS) tethered to the HMD;
perform a calibration process to generate weights with values indicating priorities using a machine learning algorithm for dynamically combining the first and the second EGT vectors based on a telemetry indicator comprising a measure of at least one of:
ambient light, head position, a user distance from the external display, or head orientation;
dynamically assign a weight with a value indicating a priority to each of the first and second EGT vectors based on telemetry data collected in varying real-time telemetry conditions;
calculate a composite EGT vector using a first weight associated with the first EGT system and a second weight associated with the second EGT system, wherein the first and second weights are calculated based on the telemetry indicator;
at least one of: (i) in response to a value increase in the telemetry indicator, increase the first weight or decrease the second weight, or (ii) in response to a value increase in the telemetry indicator, decrease the first weight or increase the second weight;
at least one of: (i) in response to a value decrease in the telemetry indicator, increase the first weight or decrease the second weight, or (ii) in response to a value decrease in the telemetry indicator, decrease the first weight or increase the second weight; and
render an image of the user's real-world environment on the HMD or the external display, in part, based upon the composite EGT vector.
2. The IHS of claim 1 , wherein the first gaze vector is defined by a pupil center and a set of glints generated in a user's eye, as detected by the first EGT system, and wherein the second gaze vector is defined by the pupil center and another set of glints generated in the user's eye, as detected by the second EGT system.
3. The IHS of claim 1 , wherein the first and second EGT systems each comprise a visual spectrum camera or a near-infrared (NIR) camera.
4. The HIS of claim 1 , wherein the program instructions, upon execution by the processor, further cause the IHS to transform the first and second EGT vectors into world space coordinates prior to calculating the composite EGT vector.
5. The HIS of claim 1 , wherein the program instructions, upon execution by the processor, further cause the IHS to store the first and second weights in a header portion of the composite EGT vector.
6. The IHS of claim 1 , wherein the first gaze vector and the second gaze vector are collected concurrently during execution of a virtual, augmented, or mixed reality (xR) application.
7. The IHS of claim 1 , wherein the program instructions, upon execution by the processor, further configure the IHS to:
receive a third EGT vector from a third EGT system; and
calculate the composite EGT vector, in part, using a third weight associated with the third EGT system.
8. A hardware memory having program instructions stored thereon for handling multiple Eye-Gaze Tracking (EGT) sources in virtual, augmented, or mixed reality applications and that, upon execution by a processor of an Information Handling System (IHS), cause the IHS to:
receive a first Eye-Gaze Tracking (EGT) vector from a first EGT system coupled to a Head-Mounted Display (HMD);
receive a second EGT vector from a second EGT system coupled to an external display of the IHS, wherein the HMD is tethered to the IHS, wherein the first and second weights are calculated based on a telemetry indicator;
perform a calibration process to generate weights with values indicating priorities using a machine learning algorithm for dynamically combining the first and the second EGT vectors based on a telemetry indicator comprising a measure of at least one of:
ambient light, head position, a user distance from the external display, or head orientation;
dynamically assign a weight with a value indicating a priority to each of the first and second EGT vectors based on the telemetry indicator;
calculate a composite EGT vector using a first weight associated with the first EGT system and a second weight associated with the second EGT system, wherein the first and second weights are calculated based on the telemetry indicator;
at least one of: (i) in response to a value increase in the telemetry indicator, increase the first weight or decrease the second weight, or (ii) in response to a value increase in the telemetry indicator, decrease the first weight or increase the second weight;
at least one of: (i) in response to a value decrease in the telemetry indicator, increase the first weight or decrease the second weight, or (ii) in response to a value decrease in the telemetry indicator, decrease the first weight or increase the second weight; and
render an image of the user's real-world environment on the HMD or the external display, in part, based upon the composite EGT vector.
9. The hardware memory of claim 8 , wherein the first gaze vector is defined by a pupil center and a set of glints generated in a user's eye, as detected by the first EGT system, and wherein the second gaze vector is defined by the pupil center and another set of glints generated in the user's eye, as detected by the second EGT system.
10. The hardware memory of claim 8 , wherein the program instructions, upon execution by the processor, further cause the IHS to transform the first and second EGT vectors into world space coordinates prior to calculating the composite EGT vector.
11. The hardware memory of claim 8 , wherein the first EGT system comprises a near-infrared (NIR) camera and wherein the second EGT system comprises a visual spectrum camera.
12. The hardware memory of claim 11 , wherein the telemetry indicator comprises a measure of IR ambient light.
13. The hardware memory of claim 8 , wherein the telemetry indicator comprises a battery level of the HMD.
14. The hardware memory of claim 8 , wherein the program instructions, upon execution by the processor, further configure the IHS to at least one of: transmit EGT seed information to the HMD via an Application Programming Interface (API) or receive EGT seed information from the HMD via the API.
15. A method for handling multiple Eye-Gaze Tracking (EGT) sources in virtual, augmented, or mixed reality applications in an Information Handling System (IHS), comprising:
receiving a first Eye-Gaze Tracking (EGT) vector from a first EGT system coupled to a Head-Mounted Display (HMD);
receiving a second EGT vector from a second EGT system coupled to an external display of the IHS, wherein the HMD is tethered to the IHS, wherein the first and second weights are calculated based on a telemetry indicator;
performing a calibration process to generate weights with values indicating priorities using a machine learning algorithm for dynamically combining the first and the second EGT vectors based on a telemetry indicator comprising a measure of at least one of: ambient light, head position, a user distance from the external display, or head orientation;
dynamically assigning a weight with a value indicating a priority to each of the first and second EGT vectors based on the telemetry indicator;
calculating a composite EGT vector using a first weight associated with the first EGT system and a second weight associated with the second EGT system, wherein the first and second weights are calculated based on the telemetry indicator;
at least one of: (i) in response to a value increase in the telemetry indicator, increasing the first weight or decreasing the second weight, or (ii) in response to a value increase in the telemetry indicator, decreasing the first weight or increasing the second weight;
at least one of: (i) in response to a value decrease in the telemetry indicator, increasing the first weight or decreasing the second weight, or (ii) in response to a value decrease in the telemetry indicator, decreasing the first weight or increasing the second weight; and
rendering an image of the user's real-world environment on the HMD or the external display, in part, based upon the composite EGT vector.