Augmented reality navigation systems for use with robotic surgical systems and methods of their use
The present disclosure is directed to augmented reality navigation systems and methods of their use that, inter alia, address the need for systems and methods of robotic surgical system navigation with reduced distraction to surgeons. Augmented reality navigation systems disclosed herein enable a surgeon to maintain focus on a surgical site and/or surgical tool being used in a surgical procedure while obtaining a wide range of navigational information relevant to the procedure. Navigational information can appear in the augmented reality navigation system as being presented on virtual displays that sit in a natural field of view of a surgeon during a procedure. Navigational information can also appear to be overlaid over a patient's anatomy. Augmented reality navigation systems comprise a head mounted display comprising an at least partially transparent display screen, at least one detector connected to the head mounted display for identifying real-world features, and a computer subsystem.
1. An augmented reality navigation system for use with a robotic surgical system, the system comprising:
a head mounted display comprising an at least partially transparent display screen configured to display augmentation graphics which appear to a user to be superimposed on at least a portion of a natural field of view of the user;
at least one detector for identifying real-world features, the at least one detector connected to the head mounted display;
a processor of a computing device; and
a non-transitory computer readable medium having instructions stored thereon, wherein the instructions, when executed by the processor, cause the processor to:
receive, by the processor, a detector input signal from the at least one detector, wherein the detector input signal corresponds to a field of view of the at least one detector and the field of view comprises at least a portion of anatomy of a patient,
determine, by the processor, one or more real-world features in the detector input signal that correspond to a pointer tool, wherein the pointer tool is used to indicate locations and/or trajectories relevant to a surgical procedure,
determine, by the processor, a position and an orientation of the pointer tool based, at least in part, on a relative position and orientation of the determined real-world features in the detector input signal, wherein the relative position and orientation are relative to a position and orientation of the head mounted display, wherein the relative position may be taken as an absolute position and/or orientation represented in a unified coordinate system that resulted from a registration of the augmented reality navigation system and the robotic surgical system,
determine and/or update, by the processor, a trajectory for use in the surgical procedure based, at least in part, on at least one of the position and the orientation of the pointer tool,
output, by the processor, the trajectory to the robotic surgical system,
wherein the augmented reality navigation system includes a gesture sensor to sense presence of a user's hand and sense movement of the user's hand.
2. The augmented reality navigation system of claim 1 , wherein the instructions cause the processor to:
render, by the processor, pointer tool augmentation graphics comprising a representation of at least a portion of the pointer tool hidden from the natural field of view of the user based on the determined position and/or the determined orientation of the pointer tool;
optionally, modify, by the processor, at least a portion of pointer tool augmentation graphics based on the relative locations and/or orientations determined from the detector input signal; and
display, by the processor, the pointer tool augmentation graphics on the display screen such that the pointer tool augmentation graphics appear overlaid on the anatomy of the patient in a location of the at least a portion of the pointer tool hidden from the natural field of view of the user.
3. The augmented reality navigation system of claim 1 , wherein the instructions cause the processor to:
render, by the processor, trajectory augmentation graphics that represent the trajectory; and
display, by the processor, on the display screen, the trajectory augmentation graphics such that the user views the trajectory augmentation graphics overlaid over the anatomy of the patient and the trajectory augmentation graphics indicate a physical trajectory that can be followed during the surgical procedure.
4. The augmented reality navigation system of claim 3 , wherein the system comprises
a motion sensor connected to the head mounted display for outputting a motion signal based on measured motion of the head mounted display; and the instructions cause the processor to:
update, by the processor, the relative position and orientation of the determined real-world features in the detector input signal based on motion detected by the motion sensor; and
update, by the processor, the trajectory augmentation graphics based on the updated relative position and orientation.
5. The augmented reality navigation system of claim 1 , wherein the instructions cause the processor to:
determine, by the processor, a relative location and/or orientation for each of at least one real-world feature from the detector input signal;
modify, by the processor, an anatomical model of a patient based on the relative location(s) and/or orientation(s) determined from the detected input signal, thereby forming an updated anatomical model;
render, by the processor, anatomical model augmentation graphics based at least in part on the updated anatomical model; and
display, by the processor, on the display screen, the anatomical model augmentation graphics such that the updated anatomical model appears overlaid on the anatomy of the patient.
6. The augmented reality navigation system of claim 1 , wherein the instructions cause the processor to:
automatically move, by the processor, a robotic arm and/or end effector of the robotic surgical system to be aligned with the trajectory.
7. The augmented reality navigation system of claim 6 , wherein the instructions cause the processor to:
automatically move, by the processor, the robotic arm and/or end effector of the robotic surgical system along the trajectory.
8. The augmented reality navigation system of claim 1 , wherein the instructions cause the processor to:
receive, by the processor, a user input signal for identifying when the pointer tool is in a desired position and/or orientation; and
determine, by the processor, the position and the orientation of the pointer tool in response to the user input signal being received.
9. The augmented reality navigation system of claim 8 , wherein the system comprises a motion sensor connected to the head mounted display for outputting a motion signal based on measured motion of the head mounted display, and the user input signal is a gesture determined based, at least in part, on motion measured by the motion sensor.
10. The augmented reality navigation system of claim 1 , wherein the instructions cause the processor to:
define and/or update, by the processor, a haptic object that comprises the trajectory; and
constrain, by the processor, motion of a robotic arm and/or end effector such that motion of at least a portion of a surgical tool attached to the robotic arm and/or end effector is constrained to within the haptic object.
11. An augmented reality navigation system for use with a robotic surgical system, the system comprising:
a head mounted display comprising an at least partially transparent display screen configured to display augmentation graphics which appear to a user to be superimposed on at least a portion of a natural field of view of the user;
at least one detector for identifying real-world features, the at least one detector connected to the head mounted display; and
a computer subsystem configured to determine (i) one or more real-world features in a detector input signal, received from the least one detector, that correspond to a pointer tool, wherein the pointer tool is used to indicate locations and/or trajectories relevant to a surgical procedure and (ii) a position and an orientation of the pointer tool based, at least in part, on the one or more real-world features, wherein a relative position of the one or more real-world features may be taken as an absolute position and/or orientation represented in a unified coordinate system that resulted from a registration of the augmented reality navigation system and the robotic surgical system,
wherein the computer subsystem is configured to determine a trajectory for use in the surgical procedure based, at least in part, on at least one of the position and the orientation of the pointer tool and output the trajectory to the robotic surgical system,
wherein the augmented reality navigation system includes a gesture sensor to sense presence of a user's hand and sense movement of the user's hand.
12. The augmented reality navigation system of claim 11 , wherein the computer subsystem is configured to display, on the display screen, pointer tool augmentation graphics comprising a representation of at least a portion of the pointer tool hidden from the natural field of view of the user based on the determined position and/or the determined orientation of the pointer tool received from the computer subsystem.
13. The augmented reality navigation system of claim 11 , wherein the computer subsystem is configured to display, on the display screen, trajectory augmentation graphics received from the computer subsystem that represent the trajectory such that the user views the trajectory augmentation graphics overlaid over an anatomy of the patient and the trajectory augmentation graphics indicate a physical trajectory that can be followed during the surgical procedure.
14. The augmented reality navigation system of claim 13 , wherein the system comprises a motion sensor connected to the head mounted display for outputting a motion signal based on measured motion of the head mounted display, wherein the computer subsystem is configured to update a relative position and orientation of the determined one or more real-world features in the detector input signal based on motion detected by the motion sensor and, subsequently, update the trajectory augmentation graphics based on the updated relative position and orientation.
15. The augmented reality navigation system of claim 11 , wherein
the computer subsystem is configured to modify an anatomical model of a patient based on one or more relative location(s) and/or orientation(s) determined from the detector input signal, thereby forming an updated anatomical model, and
the computer subsystem is configured to display, on the display screen, anatomical model augmentation graphics corresponding to the updated anatomical model such that the updated anatomical model appears overlaid on an anatomy of the patient.
16. The augmented reality navigation system of claim 11 , wherein the computer subsystem is configured to automatically move a robotic arm and/or end effector of the robotic surgical system to be aligned with the trajectory.
17. The augmented reality navigation system of claim 16 , wherein the computer subsystem is configured to automatically move the robotic arm and/or end effector of the robotic surgical system along the trajectory.
18. The augmented reality navigation system of claim 11 , wherein the computer subsystem is configured to receive a user input signal for identifying when the pointer tool is in a desired position and/or orientation and determine the position and the orientation of the pointer tool in response to the user input signal being received.
19. The augmented reality navigation system of claim 18 , wherein the system comprises a motion sensor connected to the head mounted display for outputting a motion signal based on measured motion of the head mounted display; and the user input signal is a gesture determined based, at least in part, on motion measured by the motion sensor.
20. The augmented reality navigation system of claim 11 , wherein the computer subsystem is configured to define a haptic object that comprises the trajectory and constrain motion of a robotic arm and/or end effector such that motion of at least a portion of a surgical tool attached to the robotic arm and/or end effector is constrained to within the haptic object.