Navigated surgical system with eye to XR headset display calibration
A camera tracking system for computer assisted navigation during surgery operatively determines a first pose of a second extended-reality (XR) headset relative to stereo tracking cameras located on a first XR headset based on first tracking information from the stereo tracking cameras. The camera tracking system determines a second pose of eyes of a user wearing the second XR headset relative to the stereo tracking cameras located on the first XR headset based on second tracking information from the stereo tracking cameras. The camera tracking system also calibrates an eye-to-display relationship defining pose of the eyes of the user wearing the second XR headset to a display device of the second XR headset based on the determined first and second poses. The camera tracking system also controls where symbols are displayed on the display device of the second XR headset based on the eye-to-display relationship.
1. A camera tracking system for computer assisted navigation during surgery operative to:
determine a first pose of a second extended-reality (XR) headset relative to stereo tracking cameras located on a first XR headset based on first tracking information from the stereo tracking cameras;
determine a second pose of eyes of a user wearing the second XR headset relative to the stereo tracking cameras located on the first XR headset based on second tracking information from the stereo tracking cameras;
calibrate an eye-to-display relationship defining pose of the eyes of the user wearing the second XR headset to a display device of the second XR headset based on the determined first and second poses; and
control where symbols are displayed on the display device of the second XR headset based on the eye-to-display relationship.
2. The camera tracking system of claim 1 , further operative to:
obtain an XR headset-to-display transform between a pose of the second XR headset and a pose of the display device of the second XR headset,
wherein the determination of the second pose of the eyes of the user wearing the second XR headset relative to the stereo tracking cameras located on the first XR headset is performed based on the second tracking information from the stereo tracking cameras located on the first XR headset and the XR headset-to-display transform.
3. The camera tracking system of claim 2 , wherein:
the first tracking information from the stereo tracking cameras tracks a reference array on the second XR headset;
the second tracking information from the stereo tracking cameras tracks the eyes of the user wearing the second XR headset; and
the XR headset-to-display transform relates the pose of the reference array on the second XR headset and the pose of the display device of the second XR headset.
4. The camera tracking system of claim 1 , wherein:
the determination of the first pose of the second XR headset relative to the stereo tracking cameras located on the first XR headset, includes to determine first offset distances between the stereo tracking cameras located on the first XR headset and the second XR headset;
the determination of the second pose of the eyes of the user wearing the second XR headset relative to the stereo tracking cameras located on the first XR headset, includes to determine second offset distances between the stereo tracking cameras located on the first XR headset and the eyes of the user; and
the calibration of the eye-to-display relationship defining pose of the eyes of the user wearing the second XR headset to the display device of the second XR headset, includes to determine third offset distances between the eyes of the user and the display device of the second XR headset based on the first and second offset distances.
5. The camera tracking system of claim 1 , wherein the determination of the second pose and the calibration of the eye-to-display relationship are performed responsive to detection of the eyes of the user wearing the second XR headset when imaged in video frames from the stereo tracking cameras located on the first XR headset.
6. The camera tracking system of claim 1 , wherein the second XR headset is a reflected image from a reflective surface of the first XR headset imaged in video frames from the stereo tracking cameras located on the first XR headset.
7. The camera tracking system of claim 1 , wherein:
the control of where information is displayed on the display device of the second XR headset based on the eye-to-display relationship, includes to adjust a projected image displayed on a see-through display screen of the display device of the second XR headset based on the eye-to-display relationship.
8. The camera tracking system of claim 7 , further operative to:
obtain a display-to-eye distortion transform relating optical distortion of real-world images passing through the see-through display screen of the display device of the second XR headset to where a wearer's eyes are posed relative to the see-through display screen, and
further control where symbols are displayed on the see-through display screen of the display device of the second XR headset based on the eye-to-display relationship and the display-to-eye distortion transform.
9. The camera tracking system of claim 1 , further operative to:
responsive to expiration of a threshold recalibration time since a last calibration of the eye-to-display relationship was performed, displaying a prompt on the display device of the second XR headset indicating that the user should look at the first XR headset.
10. The camera tracking system of claim 1 , further operative to:
responsive to determining the second XR headset has shifted more than a threshold amount relative to the eyes of the user wearing the second XR headset, displaying a prompt on the display device of the second XR headset indicating that the user should adjust pose of the second XR headset relative to the eyes of the user.
11. The camera tracking system of claim 1 , further operative to:
determine a third pose of the first XR headset relative to second stereo tracking cameras located on the second XR headset based on third tracking information from the second stereo tracking cameras;
determine a fourth pose of eyes of a user wearing the first XR headset relative to the second stereo tracking cameras located on the second XR headset based on fourth tracking information from the second stereo tracking cameras;
calibrate an eye-to-display relationship defining pose of the eyes of the user wearing the first XR headset to the display device of the first XR headset based on the determined third and fourth poses; and
control where symbols are displayed on the display device of the first XR headset based on the eye-to-display relationship.
12. The camera tracking system of claim 11 , further operative to:
perform time synchronization between video steams from the first XR headset and the second XR headset,
wherein the first, second, third, and fourth poses are determined using the output of the time synchronization between the video streams.
13. A computer program product comprising a non-transitory computer readable medium storing program code executable by a camera tracking system is operative to:
determine a first pose of the second extended-reality (XR) headset relative to stereo tracking cameras located on the first XR headset based on first tracking information from the stereo tracking cameras;
determine a second pose of eyes of a user wearing the second XR headset relative to the stereo tracking cameras located on the first XR headset based on second tracking information from the stereo tracking cameras;
calibrate an eye-to-display relationship defining pose of the eyes of the user wearing the second XR headset to a display device of the second XR headset based on the determined first and second poses; and
control where symbols are displayed on the display device of the second XR headset based on the eye-to-display relationship.
14. The computer program product of claim 13 , wherein the program code executable by the camera tracking system is further operative to:
obtain an XR headset-to-display transform between a pose of the second XR headset and a pose of the display device of the second XR headset,
wherein the determination of the second pose of the eyes of the user wearing the second XR headset relative to the stereo tracking cameras located on the first XR headset is performed based on the second tracking information from the stereo tracking cameras located on the first XR headset and the XR headset-to-display transform.
15. The computer program product of claim 14 , wherein:
the first tracking information from the stereo tracking cameras tracks a reference array on the second XR headset;
the second tracking information from the stereo tracking cameras tracks the eyes of the user wearing the second XR headset; and
the XR headset-to-display transform relates the pose of the reference array on the second XR headset and the pose of the display device of the second XR headset.
16. The computer program product of claim 13 , wherein:
the determination of the first pose of the second XR headset relative to the stereo tracking cameras located on the first XR headset, includes to determine first offset distances between the stereo tracking cameras located on the first XR headset and the second XR headset;
the determination of the second pose of the eyes of the user wearing the second XR headset relative to the stereo tracking cameras located on the first XR headset, includes to determine second offset distances between the stereo tracking cameras located on the first XR headset and the eyes of the user; and
the calibration of the eye-to-display relationship defining pose of the eyes of the user wearing the second XR headset to the display device of the second XR headset, includes to determine third offset distances between the eyes of the user and the display device of the second XR headset based on the first and second offset distances.
17. The computer program product of claim 13 , wherein the determination of the second pose and the calibration of the eye-to-display relationship are performed responsive to detection of the eyes of the user wearing the second XR headset when imaged in video frames from the stereo tracking cameras located on the first XR headset.
18. The computer program product of claim 13 , wherein the second XR headset is a reflected image from a reflective surface of the first XR headset imaged in video frames from the stereo tracking cameras located on the first XR headset.
19. The camera tracking system of claim 13 , wherein:
the control of where information is displayed on the display device of the second XR headset based on the eye-to-display relationship, includes to adjust a projected image displayed on a see-through display screen of the display device of the second XR headset based on the eye-to-display relationship.
20. The computer program product 19 , wherein the program code executable by the camera tracking system is further operative to:
obtain a display-to-eye distortion transform relating optical distortion of real-world images passing through the see-through display screen of the display device of the second XR headset to where a wearer's eyes are posed relative to the see-through display screen, and
further control where symbols are displayed on the see-through display screen of the display device of the second XR headset based on the eye-to-display relationship and the display-to-eye distortion transform.