Techniques for virtualized tool interaction
A computer implemented method is provided for a virtual training system. A virtual surgical simulation associated with a type of surgical procedure is accessed. Image data associated with a controller and a workspace is received. Controller data corresponding to a controller interaction is received. A first interaction of the controller within the workspace based on at least one of the image data and the controller data is determined. Using the set of one or more transformation rules, the first interaction of the controller is transformed to a manipulation of a virtualized surgical tool in the virtual surgical simulation. A representation is output of the manipulation of the virtualized surgical tool.
1. A computer implemented method comprising:
accessing a virtual surgical simulation associated with a type of surgical procedure, wherein the virtual surgical simulation includes a set of one or more transformation rules that correspond to a model for a virtualized surgical tool to be represented in the virtual surgical simulation, the virtualized surgical tool being simulated by a controller that is handheld;
receiving image data associated with the controller that is handheld and a workspace, wherein the workspace is associated with a field of view of an imaging system and/or a stereo tracker that captures the image data;
processing the image data to discard the image data without the controller in the field of view;
receiving controller data corresponding to a controller interaction, the controller data being measured by one or more sensors embedded in the controller;
determining a first interaction of the controller that is handheld within the workspace based on at least one of the image data and the controller data;
transforming, using the set of one or more transformation rules, the first interaction of the controller to a manipulation of the virtualized surgical tool in the virtual surgical simulation; and
outputting a representation of the manipulation of the virtualized surgical tool.
2. The method of claim 1 wherein the image data is generated based on projection of a structured light field.
3. The method of claim 1 further comprising projecting a light pattern that is present in the image data.
4. The method of claim 3 further comprising:
detecting a distortion in the light pattern;
determining a shape of an object in the image data based on the distortion in the light pattern; and
determining the shape of the object is the controller.
5. The method of claim 1 wherein receiving image data further comprises:
emitting infrared light; and
receiving, by an infrared stereo camera, the image data.
6. The method of claim 1 wherein the virtualized surgical tool is determined based on receiving a tool model selected by a user.
7. The method of claim 1 wherein the controller data is associated with seven degrees of freedom.
8. The method of claim 1 wherein the controller is tethered to a mobile device.
9. The method of claim 1 wherein determining the manipulation of the virtualized surgical tool further comprises determining a trajectory associated with the controller.
10. The method of claim 1 further comprising detecting one or more markers associated with the controller from the image data.
11. The method of claim 10 further comprising:
determining three or more markers are present in the image data; and
determining, using the three or more markers, a full pose of a rigid body in a workspace coordinate system corresponding to the controller.
12. The method of claim 1 wherein the virtual surgical simulation further comprises a 3D environment.
13. The method of claim 1 wherein the virtual surgical simulation is output to a display on a mobile device.
14. The method of claim 1 wherein the virtual surgical simulation is output to an augmented reality headset.
15. The method of claim 1 wherein the virtual surgical simulation is output to a virtual reality headset.
16. The method of claim 1 further comprising transmitting data to the controller via at least one of a Bluetooth™ connection and a Wi-Fi connection.
17. The method of claim 1 wherein the controller comprises gripper arms with fingertip components and a gripper motor controlled by an embedded computer of the controller, and the controller data further comprises pressure data associated with force sensors on the gripper arms of the controller.
18. The method of claim 1 , wherein the set of one or more transformation rules are updated based on one or more hand features of a user.
19. A computer-program product tangibly embodied in a non-transitory machine-readable storage medium, including instructions configured to cause one or more data processors to perform actions including:
accessing a virtual surgical simulation associated with a type of surgical procedure, wherein the virtual surgical simulation includes a set of one or more transformation rules that correspond to a model for a virtualized surgical tool to be represented in the virtual surgical simulation, the virtualized surgical tool being simulated by a controller that is handheld;
receiving image data associated with the controller and a workspace, wherein the workspace is associated with a field of view of an imaging system and/or a stereo tracker that captures the image data;
processing the image data to discard the image data without the controller in the field of view;
receiving controller data corresponding to a controller interaction, the controller data being measured by one or more sensors embedded in the controller;
determining a first interaction of the controller within the workspace based on at least one of the image data and the controller data;
transforming, using the set of one or more transformation rules, the first interaction of the controller to a manipulation of the virtualized surgical tool in the virtual surgical simulation; and
outputting a representation of the manipulation of the virtualized surgical tool.
20. A system comprising:
one or more data processors; and
a non-transitory computer readable storage medium containing instructions which when executed on the one or more data processors, cause the one or more data processors to perform actions including:
accessing a virtual surgical simulation associated with a type of surgical procedure, wherein the virtual surgical simulation includes a set of one or more transformation rules that correspond to a model for a virtualized surgical tool to be represented in the virtual surgical simulation, the virtualized surgical tool being simulated by a controller that is handheld;
receiving image data associated with a controller and a workspace, wherein the workspace is associated with a field of view of an imaging system and/or a stereo tracker that captures the image data;
processing the image data to discard the image data without the controller in the field of view;
receiving controller data corresponding to a controller interaction, the controller data being measured by one or more sensors embedded in the controller, wherein the controller comprises gripper arms with fingertip components and a gripper motor controlled by an embedded computer of the controller, and the controller data further comprises pressure data associated with force sensors on the gripper arms of the controller;
determining a first interaction of the controller within the workspace based on at least one of the image data and the controller data;
transforming, using the set of one or more transformation rules, the first interaction of the controller to a manipulation of the virtualized surgical tool in the virtual surgical simulation, and
outputting a representation of the manipulation of the virtualized surgical tool.