System and method for displaying anatomy and devices on a movable display
An image display system is provided comprised of a virtual window system that creates a visual coherency between the patient's anatomical images and the actual patient by aligning the image on the display to the patient and then presenting the image to the user in a way that feels as if the user is looking directly into the patient through the display. The image shown within the image display system is dependent upon the position of the image display apparatus and the position of the user so that the display orientation of the image may be biased slightly toward the user to improve ergonomics and usability.
1. A system for displaying an image of a tool and an image of a patient's anatomy, said system comprising:
a repositionable display screen configured to show the images of the tool and the patient's anatomy;
a robotic device configured to control movement of the tool; and
a processor configured to receive:
(a) the image of the patient's anatomy;
(b) position data and orientation data for the tool;
(c) position data for the patient's anatomy;
(d) position data for the display screen; and
(e) position data for a user's position relative to the patient,
wherein the processor is configured to:
superimpose the image of the tool on the image of the patient's anatomy and reposition the image of the patient's anatomy on the display screen in real time based on the position data for the user's position relative to the patient so the images of both the patient's anatomy and the tool are substantially aligned with the patient as the display screen is moved over the patient,
allow the user to selectively angle the aligned images away from the user so that when the display screen is angled toward the user, the aligned images will appear flat relative to the patient,
receive a user input comprising a selection of an anatomical target on the image of the patient's anatomy,
output a predicted position of the tool based on the position data for the tool and the orientation data for the tool relative to the position data for the patient's anatomy,
project a path from the predicted position of the tool to the anatomical target,
determine that the path is free of collisions (i) between the tool and an internal anatomy of the patient and (ii) between the tool and another medical device external to the patient,
cause the path and an indication that the path is collision-free to be displayed on the display screen, and
cause a collision space of the robotic device to be overlaid on the image of the patient's anatomy.
2. A system as in claim 1 , wherein the processor is configured to receive a pre-operative static image of the patient's anatomy.
3. A system as in claim 1 , wherein the processor is configured to receive a real time image of the patient's anatomy.
4. A system as in claim 3 , wherein the real time image is fluoroscopic.
5. A system as in claim 3 , wherein the real time image is a 3 dimensional point cloud of a position of the tool within the patient's anatomy.
6. A system as in claim 1 , further comprising an external position tracker configured to track a position of the tool, a position of the patient's anatomy, and a position of the display screen in a reference frame.
7. A system as in claim 6 , wherein:
the external position tracker comprises a plurality of electromagnetic sensors,
at least one of the plurality of electromagnetic sensors is present on the tool, and
at least one of the plurality of electromagnetic sensors is affixed to the patient.
8. A system as in claim 1 , further comprising an articulated support coupled to the display screen to hold the display screen over the patient, the articulated support having an encoder configured to provide the position data for the display screen to the processor.
9. A system as in claim 1 , wherein the processor is configured to allow the display screen to be repositioned relative to its associated image by: interrupting a control loop within the processor between the display screen and the associated image; freezing the associated image on the display screen at the time of the interruption; and uninterrupting the control loop between the display screen and the associated image subsequent to the display screen being repositioned to a desired position.
10. A system as in claim 1 , wherein the processor is configured to selectively decouple a relationship between the display screen and the image of the patient's anatomy displayed on the display screen based at least in part on a signal received from a user input device.
11. A system as in claim 1 , wherein the processor is configured to change a relationship between the display screen and the image of the patient's anatomy displayed on the display screen based at least in part on a signal received from a user input device.
12. A system for displaying an image of a tool and an image of a patient's anatomy, said system comprising:
a repositionable display screen configured to show the images of the tool and the patient's anatomy;
a robotic device configured to control movement of the tool; and
a processor configured to receive:
(a) the image of the patient's anatomy;
(b) position data for the patient's anatomy;
(c) position data for the display screen; and
(d) position data and orientation data for the tool;
wherein the processor is configured to:
superimpose the image of the tool on the image of the patient's anatomy and reposition the image of the patient's anatomy on the display screen in real time based on the position data for the display screen so the image of the both the patient's anatomy and the tool are substantially aligned with the patient as the display screen is moved,
output a predicted position of the tool based on the position data for the tool and the orientation data for the tool relative to the position data for the patient's anatomy,
project a path from the predicted position to an anatomical target of the patient's anatomy,
determine that the path from the predicted position to the anatomical target is free of collisions (i) between the tool and another structure of the patient's anatomy and (ii) between the tool and another medical device external to the patient,
output the path and an indication that the path from the predicted position to the anatomical target is free of collisions, and
cause a collision space of the robotic device to be overlaid on the image of the patient's anatomy.
13. A system as in claim 12 , wherein the processor is configured to track a position of the patient in real time and shift a coordinate system associated with the display screen in response to changes in position of the patient.
14. A system as in claim 12 , wherein the processor is configured to receive a real time image of the patient's anatomy.
15. A system as in claim 14 , wherein the real time image is ultrasonic.
16. A system as in claim 12 , further comprising an external position tracker configured to track a position of the patient and a position of the display screen in a reference frame.
17. A system as in claim 16 , wherein the external position tracker comprises a plurality of electromagnetic sensors.
18. A system as in claim 17 , wherein:
at least one of the plurality of electromagnetic sensors is affixed to the patient, and
at least one of the plurality of electromagnetic sensors is affixed to the display screen.
19. A system as in claim 12 , wherein the processor is configured to allow the display screen to be repositioned relative to its associated image by: interrupting a control loop within the processor between the display screen and the associated image; freezing the associated image on the display screen at the time of the interruption; and uninterrupting the control loop between the display screen and the associated image subsequent to the display screen being repositioned to a desired position.
20. A system as in claim 12 , wherein the processor is configured to selectively decouple a relationship between the display screen and an image displayed on the display screen based at least in part on a signal received from a user input device.
21. A system as in claim 12 , wherein the processor is configured to change a relationship between the display screen and an image displayed on the display screen based at least in part on a signal received from a user input device.
22. A system as in claim 12 , wherein the display screen is repositionable in a first axis with a first scaling factor for the displayed image of the patient's anatomy and repositionable in a second axis different from the first axis with a second scaling factor different from the first scaling factor for the displayed images, the first axis comprising a first translational axis or a first rotational axis, and the second axis comprising a second translational axis different from the first translational axis or a second rotational axis different from the first rotational axis.
23. A system as in claim 22 , wherein the first scaling factor is in a range between 1:1 and 1.5:1.
24. A system as in claim 22 , wherein the second scaling factor is in a range between 1:1 and 1.5:1.
25. A system as in claim 1 , wherein the processor is configured to freeze the aligned images on the display screen while the display screen is being repositioned, unfreeze the aligned images after the display screen has been repositioned, and resume the alignment of the images with the patient as the display screen is further moved over the patient.
26. A system as in claim 12 , wherein the processor is configured to freeze images on the display screen while the display screen is being repositioned, and unfreeze images on the display screen after the display screen has been repositioned.
27. A system as in claim 1 , wherein the processor is configured to cause another path from the predicted position to the selected anatomical target that includes at least one collision to be displayed.
28. A system as in claim 12 , wherein the processor is configured to:
cause another path from the predicted position to the anatomical target to be displayed on the display screen along with another indication that the another path is not free of collisions.
29. A system as in claim 1 , wherein the indication is a first indication displayed on the display screen in response to determining that the path from the predicted position to the anatomical target is collision-free, wherein the processor is further configured to display on the display screen a second indication in response to determining that the path from the predicted position to the anatomical target is not collision-free.
30. A system as in claim 12 , wherein the indication is a first indication outputted in response to determining that the path from the predicted position to the anatomical target is collision-free, wherein the processor is further configured to output a second indication in response to determining that the path from the predicted position to the anatomical target is not collision-free.
31. A system, comprising:
a display comprising a display position sensor configured to generate position data;
a medical tool configured to be inserted into a patient's anatomy, the medical tool comprising a sensor configured to generate position data and orientation data for the medical tool;
a patient reference sensor configured to generate position data for the patient's anatomy;
a robotic device configured to control movement of the medical tool; and
a processor configured to:
generate an image on the display comprising a position of the medical tool with respect to the patient's anatomy superimposed on an image of the patient's anatomy based on the position data for the medical tool and the position data for the patient's anatomy,
determine that a path from the position of the medical tool to an anatomical target is free of collisions (i) between the medical tool and an internal anatomy of the patient and (ii) between the medical tool and another medical device external to the patient, based on the orientation data of the medical tool,
cause the path to be displayed on the display,
output an indication that the path is collision-free, and
cause a collision space of the robotic device to be overlaid on the image of the patient's anatomy.
32. A system as in claim 31 , wherein the processor is further configured to:
apply a rotational bias to the image on the display toward a user based on the position data for the display and position data for a user's position relative to the patient's anatomy, and
adjust an amount of the rotational bias applied to the image in response to the display being moved rotationally with respect to the user.
33. A system as in claim 31 , wherein the image of the patient's anatomy is a pre-operative image, and wherein the image on the display further comprises an intra-operative endoscopic image of the patient's anatomy superimposed on the pre-operative image.
34. A system as in claim 31 , wherein the processor is further configured to:
detect movement of the display based on the position data for the display, and
cause re-positioning of the medical tool in response to the detected movement of the display.
35. A system as in claim 31 , wherein the processor is further configured to:
indicate that the path is collision-free by causing the path to be displayed as a green line on the display, and
indicate that a second path involves a collision by causing the second path to be displayed as a red line on the display.