Systems and methods for using scene understanding for calibrating eye tracking
View Patent ↗A system can include one or more processors that determine a first position of an object in view of a user of a head mounted display (HMD) and a first gaze direction towards the first position. The object can be detected by one or more sensors. The one or more processors detect that the user is gazing at the object. The one or more processors determine, responsive to detecting that the user is gazing at the object, a second gaze direction based on at least a second position of one or more eyes of the user of the HMD provided via an eye tracking operation of the HMD. The one or more processors calibrate, based at least on a difference between the first gaze direction and the second gaze direction, the eye tracking operation of the HMD.
1. A method for calibrating eye tracking operation, the method comprising:
determining, by one or more processors of a head mounted display (HMD), a first position of an object in view of a user of the HMD and a first predicted gaze direction towards the first position using the first position of the object, the object detected by one or more sensors;
detecting, by the one or more processors, that the user is gazing at the object;
determining, by the one or more processors responsive to detecting that the user is gazing at the object, a second gaze direction based on at least a second position of one or more eyes of the user of the HMD provided via an eye tracking operation of the HMD;
determining, by the one or more processors, a first vergence plane of the first predicted gaze direction and a second vergence plane of the second gaze direction; and
calibrating, by the one or more processors, based at least on a difference between the first vergence plane of the first predicted gaze direction and the second vergence plane of the second gaze direction, the eye tracking operation of the HMD.
2. The method of claim 1 , further comprising prompting, by the one or more processors, the user of the HMD to gaze at the detected object.
3. The method of claim 1 , further comprising determining the first position of the object responsive to detecting, by the one or more processors, one of the object coming into view of the one or more sensors or the object in a first image captured subsequent to a second image in which the object is not detected.
4. The method of claim 1 , further comprising detecting, by the one or more sensors, that the user is gazing at the object responsive to an interaction of the user with the object.
5. The method of claim 1 , further comprising detecting, by the one or more sensors, that the user is gazing at the object responsive to the user reading text associated with the object.
6. The method of claim 1 , further comprising:
detecting, by the one or more processors during use of the HMD, a plurality of objects and a plurality of gaze directions corresponding to the plurality of objects; and
calibrating, by the one or more processors, the eye tracking operation responsive to each of a plurality of differences between the plurality of gaze directions and corresponding gaze directions provided via the eye tracking operation.
7. A system, comprising:
one or more processors configured to:
determine a first position of an object in view of a user of the HMD and a first predicted gaze direction towards the first position using the first position of the object, the object detected by one or more sensors;
detect that the user is gazing at the object;
determine, responsive to detecting that the user is gazing at the object, a second gaze direction based on at least a second position of one or more eyes of the user of the HMD provided via an eye tracking operation of the HMD;
determine a first vergence plane of the first predicted gaze direction and a second vergence plane of the second gaze direction, and
calibrate, based at least on a difference between the first vergence plane of the first predicted gaze direction and the second vergence plane of the second gaze direction, the eye tracking operation of the HMD.
8. The system of claim 7 , wherein the one or more processors are further configured to prompt the user of the HMD to gaze at the detected object.
9. The system of claim 7 , wherein the one or more processors are further configured to determine the first position of the object responsive to detecting one of the object coming into view of the one or more sensors or the object in a first image captured subsequent to a second image in which the object is not detected.
10. The system of claim 7 , wherein the one or more processors are further configured to detect that the user is gazing at the object responsive to an interaction of the user with the object.
11. The system of claim 7 , wherein the one or more processors are further configured to detect that the user is gazing at the object responsive to the user reading text associated with the object.
12. The system of claim 7 , wherein the one or more processors are further configured to:
detect, during use of the HMD, a plurality of objects and a plurality of gaze directions corresponding to the plurality of objects; and
calibrate the eye tracking operation responsive to each of a plurality of differences between the plurality of gaze directions and corresponding gaze directions provided via the eye tracking operation.
13. A computer-readable medium comprising processor-executable instructions that when executed by one or more processors, cause the one or more processors to:
determine a first position of an object in view of a user of the HMD and a first predicted gaze direction towards the first position using the first position of the object, the object detected by one or more sensors;
detect that the user is gazing at the object;
determine, responsive to detecting that the user is gazing at the object, a second gaze direction based on at least a second position of one or more eyes of the user of the HMD provided via an eye tracking operation of the HMD;
determine a first vergence plane of the first predicted gaze direction and a second vergence plane of the second gaze direction; and
calibrate, based at least on a difference between the first vergence plane of the first predicted gaze direction and the second vergence plane of the second gaze direction, the eye tracking operation of the HMD.
14. The computer-readable medium of claim 13 , further comprising instructions that cause the one or more processors to prompt the user of the HMD to gaze at the detected object.
15. The computer-readable medium of claim 13 , further comprising instructions that cause the one or more processors to determine the first position of the object responsive to detecting one of the object coming into view of the one or more sensors or the object in a first image captured subsequent to a second image in which the object is not detected.
16. The computer-readable medium of claim 13 , further comprising instructions that cause the one or more processors to detect that the user is gazing at the object responsive to detecting at least one of an interaction of the user with the object or the user reading text associated with the object.
17. The computer-readable medium of claim 13 , further comprising instructions that cause the one or more processors to:
detect, during use of the HMD, a plurality of objects and a plurality of gaze directions corresponding to the plurality of objects; and
calibrate the eye tracking operation responsive to each of a plurality of differences between the plurality of gaze directions and corresponding gaze directions provided via the eye tracking operation.