Pose measurement chaining for extended reality surgical navigation in visible and near infrared spectrums
A surgical system includes a camera tracking system that determines a first pose transform between a first object coordinate system and the first tracking camera coordinate system based on first object tracking information from the first tracking camera which indicates pose of the first object. The camera tracking system determines a second pose transform between the first object coordinate system and the second tracking camera coordinate system based on first object tracking information from the second tracking camera indicating pose of the first object, and determines a third pose transform between a second object coordinate system and the second tracking camera coordinate system based on second object tracking information from the second tracking camera indicating pose of the second object. The camera tracking system determines a fourth pose transform between the second object coordinate system and the first tracking camera coordinate system.
1. A surgical system comprising:
a camera tracking system configured to
receive tracking information related to tracked objects from a first tracking camera and a second tracking camera during a surgical procedure,
determine a first pose transform between a first object coordinate system and the first tracking camera coordinate system based on first object tracking information from the first tracking camera which indicates pose of the first object,
determine a second pose transform between the first object coordinate system and the second tracking camera coordinate system based on first object tracking information from the second tracking camera which indicates pose of the first object,
determine a third pose transform between a second object coordinate system and the second tracking camera coordinate system based on second object tracking information from the second tracking camera which indicates pose of the second object, and
determine a fourth pose transform between the second object coordinate system and the first tracking camera coordinate system based on combining the first, second, and third pose transforms.
2. The surgical system of claim 1 , wherein the camera tracking system is further configured to determine the fourth pose transform between the second object coordinate system and the first tracking camera coordinate system without use of any tracking information from the first tracking camera indicating pose of the second object.
3. The surgical system of claim 1 , wherein the camera tracking system is further configured to determine pose of the second object in the first tracking camera coordinate system based on processing through the fourth pose transform the tracking information from the first tracking camera which indicates pose of the first object, based on processing through the fourth pose transform the tracking information from the second tracking camera which indicates pose of the first object, and based on processing through the fourth pose transform the tracking information from the second tracking camera which indicates pose of the second object.
4. The surgical system of claim 1 , wherein the camera tracking system is further configured to determine the fourth pose transform between the second object coordinate system and the first tracking camera coordinate system based on second object tracking information from the first tracking camera which indicates pose of the second object and further based on a result of the combining of the first, second, and third pose transforms.
5. The surgical system of claim 4 , wherein the camera tracking system is further configured to determine pose of the second object based on applying a first weight to the first and second object tracking information from the first tracking camera, applying a second weight to the first and second object tracking information from the second tracking camera, and processing through the fourth pose transform the first weighted first and second object tracking information from the first tracking camera and the second weighted first and second object tracking information from the second tracking camera, wherein a value of the first weight indicates pose determination accuracy of the first tracking camera, wherein a value of the second weight indicates pose determination accuracy of the second tracking camera.
6. The surgical system of claim 5 , wherein the camera tracking system is further configured to adapt the value of the first weight over time responsive to determined changes in pose determination accuracy of the first tracking camera during operation of the camera tracking system, and to adapt the value of the second weight over time responsive to determined changes in pose determination accuracy of the second tracking camera during operation of the camera tracking system.
7. The surgical system of claim 6 , wherein the camera tracking system is configured to:
decrease the value of the first weight based on determining the pose determination accuracy of the first tracking camera has decreased; and
increase the value of the first weight based on determining the pose determination accuracy of the first tracking camera has increased.
8. The surgical system of claim 5 , wherein the camera tracking system is further configured to adapt the value of at least one of the first and second weights over time responsive to determined changes in pose determination accuracy of the first tracking camera relative to pose determination accuracy of the second tracking camera during operation of the camera tracking system.
9. The surgical system of claim 1 , further comprising an extended reality (XR) headset including the first tracking camera and a see-through display screen, the XR headset being configured to be worn by a user during the surgical procedure, the camera tracking system is further configured to:
display on the see-through display screen of the XR headset an XR image having a pose that is determined based on the fourth pose transform.
10. The surgical system of claim 9 , wherein the camera tracking system is further configured to:
generate the XR image as a graphical representation of the second object that is posed on the see-through display screen based on processing through the fourth pose transform the first object tracking information from the first and second tracking cameras and the second object tracking information from the second tracking camera.
11. The surgical system of claim 9 , wherein the camera tracking system comprises a navigation controller communicatively connected to the first and second tracking cameras to receive the tracking information and configured to perform the determination of the first, second, third, and fourth pose transforms.
12. The surgical system of claim 1 , wherein:
dynamic reference arrays including spaced apart fiducials are attached to each of the first object, the second object, the first tracking camera, and the second tracking camera; and
the camera tracking system is further configured, while a first partially hidden fiducial of the dynamic reference array attached to the second object can be viewed by the second tracking camera but cannot be viewed by the first tracking camera, to determine pose of the first partially hidden fiducial relative to the fiducials attached to the first object based on a combination of a determined pose of the first partially hidden fiducial relative to the fiducials attached to the second tracking camera and a determined pose of the fiducials attached to the second tracking camera relative to the fiducials attached to the first object.
13. The surgical system of claim 12 , wherein:
the camera tracking system is further configured, while a second partially hidden fiducial of the dynamic reference array attached to the second object can be viewed by the second tracking camera but cannot be viewed by the first tracking camera, to determine pose of the second partially hidden fiducial relative to the fiducials attached to the first object based on a combination of a determined pose of the second partially hidden fiducial relative to the fiducials attached to the second tracking camera and a determined pose of the fiducials attached to the second tracking camera relative to the fiducials attached to the first object.
14. The surgical system of claim 12 , wherein:
the camera tracking system is further configured, while a third partially hidden fiducial of the dynamic reference array attached to the second object cannot be viewed by the second tracking camera but can be viewed by the first tracking camera, to determine pose of the third partially hidden fiducial relative to the fiducials attached to the first object based on a combination of a determined pose of the third partially hidden fiducial relative to the fiducials attached to the first tracking camera and a determined pose of the fiducials attached to the first tracking camera relative to the fiducials attached to the first object.
15. The surgical system of claim 14 , wherein the camera tracking system is further configured to process the determined pose of the first partially hidden fiducial relative to the fiducials attached to the first object and to process the determined pose of the third partially hidden fiducial relative to the fiducials attached to the first object through a pose recovery operation to determine pose of the second object relative to the first object.
16. The surgical system of claim 14 , wherein:
the camera tracking system is further configured, while a fourth partially hidden fiducial of the dynamic reference array attached to the second object cannot be viewed by the second tracking camera but can be viewed by the first tracking camera, to determine pose of the fourth partially hidden fiducial relative to the fiducials attached to the first object based on a combination of a determined pose of the fourth partially hidden fiducial relative to the fiducials attached to the first tracking camera and a determined pose of the fiducials attached to the first tracking camera relative to the fiducials attached to the first object.
17. A method by a camera tracking system comprising:
receiving tracking information related to tracked objects from a first tracking camera and a second tracking camera during a surgical procedure,
determining a first pose transform between a first object coordinate system and the first tracking camera coordinate system based on first object tracking information from the first tracking camera which indicates pose of the first object,
determining a second pose transform between the first object coordinate system and the second tracking camera coordinate system based on first object tracking information from the second tracking camera which indicates pose of the first object,
determining a third pose transform between a second object coordinate system and the second tracking camera coordinate system based on second object tracking information from the second tracking camera which indicates pose of the second object, and
determining a fourth pose transform between the second object coordinate system and the first tracking camera coordinate system based on combining the first, second, and third pose transforms.
18. The method of claim 17 , further comprising determining pose of the second object in the first tracking camera coordinate system based on processing through the fourth pose transform the tracking information from the first tracking camera which indicates pose of the first object, based on processing through the fourth pose transform the tracking information from the second tracking camera which indicates pose of the first object, and based on processing through the fourth pose transform the tracking information from the second tracking camera which indicates pose of the second object.
19. A computer program product comprising:
a non-transitory computer readable medium storing program code executable by at least one processor of a camera tracking system to
receive tracking information related to tracked objects from a first tracking camera and a second tracking camera during a surgical procedure,
determine a first pose transform between a first object coordinate system and the first tracking camera coordinate system based on first object tracking information from the first tracking camera which indicates pose of the first object,
determine a second pose transform between the first object coordinate system and the second tracking camera coordinate system based on first object tracking information from the second tracking camera which indicates pose of the first object,
determine a third pose transform between a second object coordinate system and the second tracking camera coordinate system based on second object tracking information from the second tracking camera which indicates pose of the second object, and
determine a fourth pose transform between the second object coordinate system and the first tracking camera coordinate system based on combining the first, second, and third pose transforms.
20. The computer program product of claim 19 , wherein the program code is further executable by the at least one processor of the camera tracking system to determine pose of the second object in the first tracking camera coordinate system based on processing through the fourth pose transform the tracking information from the first tracking camera which indicates pose of the first object, based on processing through the fourth pose transform the tracking information from the second tracking camera which indicates pose of the first object, and based on processing through the fourth pose transform the tracking information from the second tracking camera which indicates pose of the second object.