Systems and methods for identifying and tracking physical objects during a robotic surgical procedure
Surgical systems and methods for preventing collision of a cutting instrument with a retractor. A robotic device includes a robotic arm to support and move the cutting instrument. A vision device is attached to the robotic device or the cutting instrument such that the vision device is movable by the robotic device. A control system controls the robotic device to move the cutting instrument along a tool path to remove material from a bone. The control system acquires vision data sets from the vision device in response to movement of the cutting instrument along the tool path and detects the retractor from the vision data sets. The control system generates an output to prevent collision of the cutting instrument with the retractor, such as, for example, defining a “no-fly” zone for the retractor and controlling the robotic device to avoid the cutting instrument from contacting the “no-fly” zone.
1 . A surgical system for removing material from a bone at a surgical site including a retractor, the surgical system comprising:
a robotic device comprising a robotic arm configured to support and move a cutting instrument along a tool path to remove the material from the bone;
a vision device attached to the robotic arm or the cutting instrument such that the vision device has a field of view including the surgical site and such that movement of the vision device relative to the surgical site occurs as a result of movement of the cutting instrument along the tool path to remove the material from the bone;
a tracker coupled to the robotic device or the cutting instrument;
a localizer configured to detect poses of the tracker; and
a control system coupled to the localizer, the robotic device and the vision device and being configured to:
control the robotic device to move the cutting instrument along the tool path to remove the material from the bone;
acquire, from the vision device, vision data sets captured from multiple perspectives of the surgical site enabled by the vision device being moved in a plurality of degrees of freedom as a result of the movement of the cutting instrument along the tool path to remove the material from the bone;
track a pose of the vision device at each location of the vision device in which the vision data sets are generated based on the detected poses of the tracker and a known spatial relationship between the tracker and the vision device;
detect the retractor from the vision data sets based on the pose of the vision device at each location of the vision device in which the vision data sets are generated; and
generate an output to prevent collision of the cutting instrument with the retractor.
2 . The surgical system of claim 1 , wherein the control system is configured to associate a virtual object with the retractor.
3 . The surgical system of claim 2 , wherein the virtual object comprises a virtual boundary defining a region to be avoided by the cutting instrument.
4 . The surgical system of claim 2 , wherein the control system generates the output by being configured to control the robotic device to limit movement of the cutting instrument relative to the virtual object.
5 . The surgical system of claim 2 , wherein the control system generates the output by being configured to control the robotic device to stop movement of the cutting instrument prior to the cutting instrument reaching the virtual object.
6 . The surgical system of claim 2 , wherein the control system associates the virtual object with the retractor by being configured to:
identify one or more features of the retractor in the vision data sets, the one or more features coded with information relating to the retractor; and
retrieve, from a non-transitory memory, the virtual object to be associated with the retractor based on the coded information of the one or more features of the retractor identified in the vision data sets.
7 . The surgical system of claim 6 , wherein the coded information comprises one or more of: a type of the retractor, a size of the retractor, dimensions of the retractor, a part number of the retractor, and a manufacturer of the retractor.
8 . The surgical system of claim 2 , wherein the virtual object comprises a 3-D model of the retractor.
9 . The surgical system of claim 1 , wherein the control system is configured to generate the output in response to determining that the retractor interferes with the tool path.
10 . The surgical system of claim 9 , wherein the control system determines that the retractor interferes with the tool path by being configured to:
associate a virtual object with the retractor; and
determine that the virtual object interferes with the tool path.
11 . The surgical system of claim 9 , wherein the control system is configured to generate the output by being configured to adjust the tool path.
12 . The surgical system of claim 1 , wherein the control system is configured to:
control the robotic device in an autonomous mode whereby the robotic device autonomously moves the cutting instrument along the tool path to remove the material from the bone, wherein the tool path is a predetermined tool path; and
acquire the vision data sets from the vision device in response to autonomous movement of the cutting instrument along the predetermined tool path.
13 . The surgical system of claim 12 , wherein the control system is configured to:
automatically detect the retractor from the vision data sets acquired in response to autonomous movement of the cutting instrument along the predetermined tool path;
automatically determine that the retractor interferes with the predetermined tool path; and
generate the output to prevent collision of the cutting instrument with the retractor by being configured to automatically adjust the predetermined tool path.
14 . The surgical system of claim 1 , wherein the control system is configured to:
control the robotic device in a manual mode whereby the robotic device moves the cutting instrument along the tool path to remove the material from the bone in response to manual manipulation of the cutting instrument by a user, wherein the tool path is a user-defined tool path determined by manual manipulation of the cutting instrument by the user; and
acquire the vision data sets from the vision device in response to movement of the cutting instrument along the user-defined tool path; and
automatically detect the retractor from the vision data sets acquired in response to movement of the cutting instrument along the user-defined tool path.
15 . The surgical system of claim 14 , wherein, in the manual mode, the control system is configured to generate the output to prevent collision of the cutting instrument with the retractor by being configured to:
associate a virtual boundary with the retractor, wherein the virtual boundary defines a region to be avoided by the cutting instrument; and
perform one or more of the following:
control the robotic device to provide haptic feedback to a user through the cutting instrument to prevent interaction of the cutting instrument with the virtual boundary;
control the robotic device to limit movement of the cutting instrument relative to the virtual boundary; and/or
control the robotic device to stop movement of the cutting instrument prior to the cutting instrument reaching the virtual boundary.
16 . The surgical system of claim 1 , further comprising a display device and wherein the control system generates the output by being configured to present, on the display device, a notification to: indicate that the retractor interferes with the tool path and/or suggest relocation of the retractor.
17 . The surgical system of claim 6 , wherein, throughout movement of the cutting instrument along the tool path, the control system is configured to track a pose of the bone utilizing the vision data sets and track a pose of the retractor based on positions of the one or more features in the vision data sets.
18 . A method of operating a surgical system to remove material from a bone at a surgical site including a retractor, the surgical system including a robotic device comprising a robotic arm configured to support and move a cutting instrument along a tool path to remove the material from the bone, a vision device attached to the robotic arm or the cutting instrument such that the vision device has a field of view including the surgical site and such that movement of the vision device relative to the surgical site occurs as a result of movement of the cutting instrument along the tool path to remove the material from the bone, a tracker coupled to the robotic device or the cutting instrument, a localizer configured to detect poses of the tracker, and a control system coupled to the localizer, the robotic device and the vision device, the method comprising the control system:
controlling the robotic device to move the cutting instrument along the tool path for removing the material from the bone;
acquiring, from the vision device, vision data sets captured from multiple perspectives of the surgical site enabled by the vision device being moved in a plurality of degrees of freedom as a result of the from the moving the cutting instrument along the tool path to remove the material from the bone;
tracking a pose of the vision device at each location of the vision device in which the vision data sets are generated based on the detected poses of the tracker and a known spatial relationship between the tracker and the vision device;
detecting the retractor from the vision data sets based on the pose of the vision device at each location of the vision device in which the vision data sets are generated; and
generating an output for preventing collision of the cutting instrument with the retractor.
19 . The surgical system of claim 1 , wherein throughout movement of the cutting instrument along the tool path to remove the material from the bone, the control system is configured to track a pose of the cutting instrument at a first rate and track a pose of the retractor based on the vision data sets at a second rate less than the first rate.
20 . The surgical system of claim 1 , wherein the control system is configured to detect the retractor from the vision data sets based on the pose of the vision device at each location of the vision device in which the vision data sets are generated by being configured to:
identify a first feature of the retractor in a first of the vision data sets and a second feature of the retractor in a second of the vision data sets, wherein the first vision data set does not include the second feature and the second vision data set does not include the first feature; and
detect the retractor based on the identified first feature, the identified second feature, the pose of the vision device for the first vision data set, and the pose of the vision device for the second vision data set.