System and methods for tele-collaboration in minimally invasive surgeries
Disclosed is an immersive, augmented reality-based, enabling technology for tele-collaboration between a local surgeon and a remote surgeon during an MIS. The technology would provide realistic visual-cues to the local surgeon for the required movement of an actuated, high degree-of-freedom surgical tool during an MIS.
1. A method comprising:
connecting a local workstation and a remote workstation;
providing to at least one of the local workstation or the remote workstation at least one of an instrument state or a scope state
and at least one of a trocar, a trocar tracking frame attached to the trocar, a scope, or a scope tracking frame attached to the scope; and
continuously updating at least one of a surgical state, a tooltip pose, data to be communicated over network, or a rendered object on a visualization screen in each of the local and remote workstations;
wherein the scope state comprises at least one of the scope's field of view (FOV), the scope's angulation, and transformation between MScope(t) and MScopeCamera(t),
wherein MScope(t) represents a pose of the scope tracking frame attached to the scope in form of 4×4 homogenous transformation matrix for time instant “t,” and
MScopeCamera(t) represents a pose of scope camera is represented by 4×4 homogenous transformation matrix at time instant “t”.
2. The method of claim 1 comprising providing the trocar and further comprising providing a label indicating a position of the trocar.
3. The method of claim 2 further comprising mapping at least one of a instrument type or a human computer interface to the label.
4. The method of claim 3 comprising mapping the human computer interface to the label.
5. The method of claim 4 further comprising interacting with the human computer interface and updating the tooltip pose of a rendered augmented tool on both the local and remote workstations.
6. The method of claim 1 , wherein the instrument state comprises a list of instruments to be used.
7. The method of claim 1 , wherein the at least one of the instrument state and the scope state is shared by both the local workstation and the remote workstation.
8. The method of claim 1 comprising rendering an augmented-reality scene on a visualization screen.
9. A system comprising:
a local system comprising
an input/output device selected from the group consisting of a microphone, a speaker, a first visualization screen, and combinations thereof,
a scope system comprising at least one of a scope, a camera, a camera system, a scope's tracking frame, and combinations thereof,
an optical tracking system,
a trocar system comprising at least one of a trocar, a trocar's tracking frame, and combinations thereof; and
a remote system connected to the operating room system via a network, the remote system comprising
a human computer interface system comprising at least one of a camera, a sensor, a user interface, and combinations thereof,
a second visualization screen.
10. The system of claim 9 , wherein the local system further comprises an operating instrument.
11. A method for remote collaboration and training, the method comprising:
transforming a hand gesture of a first user into a virtual tooltip movement;
superimposing the virtual tooltip movement on a second user's view of a visual field;
receiving a video frame;
extracting an actual tooltip from the video frame to form the virtual tooltip;
computing a position of the actual tooltip;
calibrating the position of the virtual tooltip from the hand gesture with the actual tooltip from the video stream; and
rendering a complete virtual tool if the actual tooltip and the virtual tooltip are aligned, or rendering only the virtual tooltip if the actual tooltip and the virtual tooltip are not aligned.
12. The method of claim 11 , wherein transforming the hand gesture of the first user into the virtual tooltip movement comprises
extracting a position of at least one optical marker attached to a grasper in the first user's hand and
triangulating the position into a position of the virtual tooltip.
13. The method of claim 11 comprising rendering the virtual tooltip movement generated by the first user along with a video stream from a scope's camera on a visualization screen.
14. The method of claim 11 comprising transmitting a live video stream from the first user's workstation to the second user's workstation over a network.
15. A system for remote collaboration and training, the system comprising:
a first computing system comprising first I/O devices configured for a first user to receive and send information;
a second computing system comprising second I/O devices for a second user to receive and send information,
wherein the first and second I/O devices are each selected from the group consisting of an infrared camera configured to capture the second user's hand gestures holding an instrument, the instrument, a scope configured to capture a video of a visual field of the first user, a first visualization screen configured to display the video of the visual field, a second visualization screen configured to display an augmented visual field, and combinations thereof;
a module configured to operate on at last one of the first or second computing systems, wherein the module is selected from the group consisting of
a video processing module configured to receive a video frame from a network module, extract an actual tooltip from the video frame, and compute a position of the tooltip,
a control logic module configured to take a first input from the video processing module and a reconstruction module and provide a second input to an augmentation module on graphical rendering;
an augmentation module configured to render an augmented-reality scene on the second visualization screen,
the reconstruction module configured to transform the second user's hand gestures into movements of a virtual tooltip,
the network module configured to exchange data over a network connecting the first and second computing system, and
combinations thereof.
16. The system of claim 15 , wherein the second I/O devices comprise the infrared camera, and the instrument comprises a grasper.
17. The system of claim 16 , wherein the grasper comprises
a pinching member configured to constrain a motion of the second user's hand holding the grasper and
at least one optical marker configured to trace the motion of the second user's hand and at least one of opening or closing of the grasper in the infrared camera.
18. The system of claim 17 , wherein the pinching member is configured to constrain a motion of the second user's index finger and thumb with respect to each other.
19. The system of claim 15 , wherein the reconstruction module is configured to transform the second user's hand gestures into movements of the virtual tooltip by extracting a position of the at least one optical marker attached to the grasper and triangulating the positions into a position of the virtual tooltip.
20. The system of claim 15 , wherein the control logic module is configured to calibrate the position of the virtual tooltip from the second user's hand gestures with an actual tooltip from the video stream.
21. The system of claim 15 , wherein the augmentation module is configured to receive an input in a form of video frame from the network module and decision to render a tooltip or complete tool from the control logic module.
22. The system of claim 15 , wherein the augmentation module is configured to, based on the input, render the augmented reality scene consisting of three-dimensional computer graphics rendered on the video stream.
23. The system of claim 15 , wherein the augmentation module comprises an inverse kinematics sub-module configured to compute the position of the virtual tooltip.
24. The system of claim 23 , wherein the position of the virtual tooltip comprises at least one of a degree-of-freedoms or a base frame.
25. A method comprising:
receiving a video frame including an actual tooltip, extracting the actual tooltip from the video frame, and computing a position of the actual tooltip, by a video processing module of a computing system comprising at least one processor and a data storage device in communication with the at least one processor;
receiving an input from the video processing module and a reconstruction module and providing the input to an augmentation module on graphical rendering, by a control logic module of the computing system;
rendering, by the augmentation module of the computing system, an augmented-reality scene on a first visualization screen;
transforming a user's hand gestures into movements of a virtual tooltip, by the reconstruction module of a computing system;
exchanging data, by a network module of the computing system, over a network.
26. The method of claim 25 further comprising:
capturing, by an infrared camera, the user's hand gestures simulating the holding of the actual tooltip;
capturing, by a scope, a video of a visual field; and
displaying the video of the visual field on the first visualization screen.
27. The method of claim 25 further comprising:
calibrating the position of the virtual tooltip from the user's hand gestures with the actual tooltip from the video frame; and
rendering a complete virtual tool if the actual tooltip and the virtual tooltip are aligned, or rendering only the virtual tooltip if the actual tooltip and the virtual tooltip are not aligned.