Augmented reality headset for a surgical robot
Disclosed is an augmented reality (AR) headset that provides a wearer with spatial, system, and temporal contextual information of a surgical robotic system to guide the wearer in configuring, operating, or troubleshooting the surgical robotic system prior to, during, or after surgery. The spatial context information may be rendered to display spatially-fixed 3D-generated virtual models of the robotic arms, instruments, bed, and other components of the surgical robotic system that match the actual position or orientation of the surgical robotic system in the AR headset's coordinate frame. The AR headset may communicate with the surgical robotic system to receive real-time state information of the components of the surgical robotic system. The AR headset may use the real-time state information to display context-sensitive user interface information such as tips, suggestions, visual or audio cues on maneuvering the robotic arms and table to their target positions and orientations or for troubleshooting purpose.
1. An augmented reality (AR) device for a surgical robotic system, comprising:
a sensor configured to capture image data of a first component of the surgical robotic system;
a processor configured to:
establish a global coordinate frame for the AR device and the surgical robotic system based on the image data for virtual components of the surgical robotic system to be created in the coordinate frame of the AR device;
receive spatial information and real-time system state information of the surgical robotic system; and
create a three-dimensional (3D) virtual model of a second component of the surgical robotic system in the coordinate frame of the AR device based on the spatial information and the real-time system state information of the surgical robotic system and based on the global coordinate frame; and
a display configured to present the 3D virtual model of the second component of the surgical robotic system.
2. The AR device of claim 1 , wherein the processor configured to establish the global coordinate frame for the AR device and the surgical robotic system comprises:
transmit, by the AR device, the image data of the first component to the surgical robotic system for the surgical robotic system to establish the global coordinate frame for the coordinate frame of the AR device and a coordinate frame of the surgical robotic system.
3. The AR device of claim 2 , wherein the spatial information of the surgical robotic system is received in the coordinate frame of the AR device.
4. The AR device of claim 1 , wherein the first component of the surgical robotic system is a fiduciary marker of the surgical robotic system, and wherein the processor configured to establish the global coordinate frame for the AR device and the surgical robotic system comprises:
analyze the image data of the fiduciary marker based on a fixed location of the fiduciary marker in a coordinate frame of the surgical robotic system to establish the global coordinate frame for the coordinate frame of the AR device for the coordinate frame of the surgical robotic system.
5. The AR device of claim 4 , wherein the spatial information of the surgical robotic system comprises spatial information of the second component in the coordinate frame of the surgical robotic system, and wherein the processor configured to create the 3D virtual model of the second component of the surgical robotic system in the coordinate frame of the AR device comprises:
translate the spatial information of the second component in the coordinate frame of the surgical robotic system to the coordinate frame of the AR device based on the global coordinate frame.
6. The AR device of claim 1 , further comprising the processor configured to maintain a position and orientation of the 3D virtual model of the second component of the surgical robotic system as the coordinate frame of the AR device changes relative to a coordinate frame of the surgical robotic system.
7. The AR device of claim 1 , wherein the spatial information of the surgical robotic system comprises spatial information of the second component, and wherein the processor configured to create the 3D virtual model of the second component comprises:
create a position and orientation of the second component of the surgical robotic system in the coordinate frame of the AR device based on the spatial information of the second component, wherein the position and orientation of the second component in the coordinate frame of the AR device matches an actual position and orientation of the second component or a target position and orientation of the second component.
8. The AR device of claim 7 , wherein the real-time system state information of the surgical robotic system comprises real-time system state information of the second component, and wherein the processor configured to create the 3D virtual model of the second component further comprises:
create context-sensitive information of the second component of the surgical robotic system based on the real-time system state information of the second component.
9. The AR device of claim 1 , wherein the second component of the surgical robotic system comprises a robotic arm or an operating table of the surgical robotic system.
10. The AR device of claim 9 , wherein the 3D virtual model of the second component comprises:
a 3D virtual rendering of a position and orientation of the robotic arm or the operating table; and
visual or audible communication of the real-time system state information of the robotic arm or the operating table.
11. A method comprising:
capturing, using a sensor of an augmented reality (AR) device, image data of a first component of a surgical robotic system;
establishing a global coordinate frame for the AR device and the surgical robotic system based on the image data for virtual components of the surgical robotic system to be created in the coordinate frame of the AR device;
receiving spatial information and real-time system state information of the surgical robotic system;
creating a three-dimensional (3D) virtual model of a second component of the surgical robotic system in the coordinate frame of the AR device based on the spatial information and the real-time system state information of the surgical robotic system and based on the global coordinate frame; and
presenting, on a display of the AR device, the 3D virtual model of the second component of the surgical robotic system.
12. The method of claim 11 , wherein establishing the global coordinate frame for the AR device and the surgical robotic system comprises transmitting, by the AR device, the image data of the first component to the surgical robotic system for the surgical robotic system to establish the global coordinate frame for the coordinate frame of the AR device and a coordinate frame of the surgical robotic system.
13. The method of claim 12 , wherein the spatial information of the surgical robotic system is received in the coordinate frame of the AR device.
14. The method of claim 11 ,
wherein the first component of the surgical robotic system is a fiduciary marker of the surgical robotic system, and
wherein establishing the global coordinate frame for the AR device and the surgical robotic system comprises analyzing the image data of the fiduciary marker based on a fixed location of the fiduciary marker in a coordinate frame of the surgical robotic system to establish the global coordinate frame for the coordinate frame of the AR device for the coordinate frame of the surgical robotic system.
15. The method of claim 14 ,
wherein the spatial information of the surgical robotic system comprises spatial information of the second component in the coordinate frame of the surgical robotic system, and
wherein creating the 3D virtual model of the second component of the surgical robotic system in the coordinate frame of the AR device comprises translating the spatial information of the second component in the coordinate frame of the surgical robotic system to the coordinate frame of the AR device based on the global coordinate frame.
16. The method of claim 11 further comprising maintaining a position and orientation of the 3D virtual model of the second component of the surgical robotic system as the coordinate frame of the AR device changes relative to a coordinate frame of the surgical robotic system.
17. The method of claim 11 , wherein the second component of the surgical robotic system comprises a robotic arm or an operating table of the surgical robotic system.
18. The method of claim 17 , wherein the 3D virtual model of the second component comprises:
a 3D virtual rendering of a position and orientation of the robotic arm or the operating table; and
visual or audible communication of the real-time system state information of the robotic arm or the operating table.
19. A surgical robotic system comprising:
a first component and a second component;
an augmented reality (AR) device that comprises a sensor configured to capture image data of the first component, and a display;
one or more processors; and
memory having instructions stored therein which when executed by the one or more processors causes the surgical robotic system to:
establish a global coordinate frame for the AR device and the surgical robotic system based on the image data for virtual components of the surgical robotic system to be created in the coordinate frame of the AR device;
receive spatial information and real-time system state information of the surgical robotic system; and
create a three-dimensional (3D) virtual model of the second component in the coordinate frame of the AR device based on the spatial information and the real-time system state information of the surgical robotic system and based on the global coordinate frame,
wherein the display is configured to present the 3D virtual model of the second component.
20. The surgical robotic system of claim 19 , wherein the first component comprises a fiduciary marker, and the second component comprises a robotic arm or an operating table of the surgical robotic system.