Graphical user interface for use in a surgical navigation system with a robot arm
A surgical navigation system includes: a tracker ( 125 ) for real-time tracking of a position and orientation of a robot arm ( 191 ); a source of a patient anatomical data ( 163 ) and a robot arm virtual image ( 166 ); a surgical navigation image generator ( 131 ) generating a surgical navigation image ( 142 A) including the patient anatomy ( 163 ) and the robot arm virtual image ( 166 ) in accordance to the current position and/or orientation data provided by the tracker ( 125 ); a 3D display system ( 140 ) showing the surgical navigation image ( 142 A).
1. A surgical navigation system, comprising:
a three-dimensional (3D) display system;
a tracker configured to track a position and orientation of a robot arm, a head of a surgeon, the 3D display system, a surgical instrument, and a patient anatomy in a surgical field to provide current position and orientation data of the robot arm, the head of the surgeon, the 3D display system, the surgical instrument, and the patient anatomy;
a source of a patient anatomical data, a surgical instrument model, and a robot arm data; and
a surgical navigation image generator configured to generate a surgical navigation image comprising a virtual representation of at least a portion of the patient anatomy, a virtual representation of at least a portion of the surgical instrument, and a virtual representation of at least a portion of the robot arm based on the current position and orientation data provided by the tracker and using the patient anatomical data, the surgical instrument model, and the robot arm data;
the 3D display system configured to display the surgical navigation image to the surgeon such that the virtual representation of the portion of the patient anatomy is collocated with the patient anatomy in the surgical field, the virtual representation of the portion of the surgical instrument is collocated with the surgical instrument, and the virtual representation of the portion of the robot arm is collocated with the robot arm.
2. The system of claim 1 , wherein the 3D display system is configured to selectively show or hide the display of the virtual representation of the portion of the robot arm.
3. The system of claim 1 , wherein the 3D display system is configured to adjust an opacity of the display of the virtual representation of the portion of the robot arm.
4. The system of claim 1 , wherein:
the patient anatomical data comprises a 3D reconstruction of a segmented model of the patient anatomy comprising at least two sections representing parts of the patient anatomy; and
the 3D display system is configured to adjust the display of the portion of the patient anatomy such that at least one section of the patient anatomy is displayed while at least one other section of the patient anatomy is not displayed.
5. The system of claim 1 , further comprising:
a source of an operative plan;
wherein the surgical navigation image further comprises surgical guidance indicating at least one of a trajectory or a placement of the surgical instrument according to the operative plan.
6. The system of claim 5 , wherein the surgical guidance indicates a suggested position and orientation of the surgical instrument according to the operative plan.
7. The system of claim 6 , wherein the surgical guidance includes a graphical cue indicating a required change of a current position and orientation of the surgical instrument to match the suggested position and orientation according to the operative plan.
8. The system of claim 1 , wherein the surgical navigation image further comprises a set of orthogonal or arbitrary planes of the patient anatomy.
9. The system of claim 1 , wherein the 3D display system is configured to display the surgical navigation image at a see-through device, and wherein the tracker is configured to track a position and orientation of the see-through device such that the surgical navigation image being displayed at the see-through device has the virtual representation of the portion of the patient anatomy collocated with the patient anatomy in the surgical field, the virtual representation of the portion of the surgical instrument collocated with the surgical instrument, and the virtual representation of the portion of the robot arm collocated with the robot arm underneath the see-through device when viewed by the surgeon looking from above the see-through device towards the surgical field.
10. The system of claim 1 , wherein the patient anatomical data comprises output data of a semantic segmentation process of a set of two-dimensional (2D) images of the patient anatomy.
11. The system of claim 1 , further comprising a convolutional neural network system configured to perform the semantic segmentation process to generate the patient anatomical data.
12. A method for providing an augmented reality image during an operation, comprising:
obtaining a source of a patient anatomical data, a surgical instrument model, and a robot arm data;
tracking a position and orientation of a robot arm, a head of a surgeon, a three-dimensional (3D) display system, and a patient anatomy in a surgical field to provide current position and orientation data of the robot arm, the head of the surgeon, the 3D display system, the surgical instrument, and the patient anatomy;
generating, by a surgical navigation image generator, a surgical navigation image comprising a virtual representation of at least a portion of the patient anatomy, a virtual representation of at least a portion of the surgical instrument, and a virtual representation of at least a portion of the robot arm based on the current position and orientation data provided by the tracker and using the patient anatomical data, the surgical instrument model, and the robot arm data; and
displaying the surgical navigation image at a 3D display system such that the virtual representation of the portion of the patient anatomy is collocated with the patient anatomy in the surgical field, the virtual representation of the portion of the surgical instrument is collocated with the surgical instrument, and the virtual representation of the portion of the robot arm is collocated with the robot arm.
13. The method of claim 12 , further comprising selectively showing or hiding at the 3D display system the display of the virtual representation of the portion of the robot arm.
14. The method of claim 12 , further comprising adjusting at the 3D display system an opacity of the display of the virtual representation of the portion of the robot arm.
15. The method of claim 12 , wherein the patient anatomical data comprises a 3D reconstruction of a segmented model of the patient anatomy comprising at least two sections representing parts of the patient anatomy, the method further comprising:
adjusting at the 3D display system the display of the portion of the patient anatomy such that at least one section of the patient anatomy is displayed while at least one other section of the patient anatomy is not displayed.
16. The method of claim 12 , further comprising obtaining a source of an operative plan,
wherein the surgical navigation image further comprises surgical guidance indicating at least one of a trajectory or a placement of the surgical instrument according to the operative plan.
17. The method of claim 16 , wherein the surgical guidance indicates a suggested position and orientation of the surgical instrument according to the operative plan.
18. The method of claim 17 , wherein the surgical guidance includes a graphical cue indicating a required change of a current position and orientation of the surgical instrument to match the suggested position and orientation according to the operative plan.
19. The method of claim 17 , wherein the surgical navigation image further comprises a set of orthogonal or arbitrary planes of the patient anatomy.
20. The method of claim 12 , wherein the position and the orientation of the robot arm is tracked by a tracker that tracks a marker array attached to a portion of the robot arm and by encoders positioned at one or more joints of the robot arm.