Robotic surgical collision detection systems
Systems and methods for surgical robotic collision detection in accordance with aspects of the present disclosure are disclosed. In various embodiments, a system for surgical robotic collision detection includes a robotic cart having a robotic arm, an imaging device supported by the robotic cart or the robotic arm, the imaging device captures images within a field of vision of the imaging device, and a controller in operable communication with the robotic arm and the imaging device. The controller includes a processor and a memory storing instructions which, when executed by the processor, causes the controller to: receive the images from the imaging device, generate a grid including a first plurality of spatial points from the images, and detect a potential collision within the field of vision based on the generated grid.
1. A surgical robotic collision detection system, comprising:
a robotic cart having a robotic arm;
a plurality of imaging devices supported by the robotic cart or the robotic arm, the plurality of imaging devices configured to capture images within a field of vision of the plurality of imaging devices; and
a controller in operable communication with the robotic arm and the plurality of imaging devices, the controller having a processor and a memory storing instructions thereon which, when executed by the processor, causes the controller to:
receive the images from the plurality of imaging devices;
generate a three-dimensional grid including a first plurality of spatial points from the images; and
detect a potential collision of objects located within the field of vision based on:
the generated three-dimensional grid;
the geometric orientation of the objects located within the field of vision; and
a sweep volume of movement of each of the objects located within the field of vision,
wherein any overlap in the sweep volumes of the objects located within the field of vision provides an indication of the potential collision.
2. The system according to claim 1 , wherein at least one imaging device of the plurality of imaging devices is selected from the group consisting of a stereoscopic imaging device, an optical imaging device, a ranging laser device, and an infrared (IR) imaging device.
3. The system according to claim 1 , wherein at least one imaging device of the plurality of imaging devices includes a sensor configured to capture a first image at a first time point, the first image including a first object of the objects located within the field of vision in positional relation to the sensor.
4. The system according to claim 3 , wherein the memory stores instructions which, when executed by the processor, causes the controller to receive the first image and generate a first depth map based on the first image.
5. The system according to claim 4 , wherein the controller generates a first point cloud based on the first depth map, the first point cloud including the first plurality of spatial points contained within the grid.
6. The system according to claim 5 , wherein the controller segments the first plurality of spatial points to identify a first spatial point subset of the first point cloud, each spatial point in the first spatial point subset corresponds to a surface of the first object.
7. The system according to claim 6 , where the memory includes instructions that, when executed by the processor, causes the controller to:
compare the first spatial point subset to a pre-identified configuration of a structure of the first object to identify the first object within the field of vision of the plurality of imaging devices.
8. The system according to claim 3 , wherein:
the sensor of the at least one imaging device of the plurality of imaging devices captures a second image at a second time point, and
the memory further includes instructions that, when executed by the processor, causes the controller to:
receive the second image; and
generate a second depth map.
9. The system according to claim 8 , where the memory further includes instructions that, when executed by the at least one processor, causes the controller to:
generate a second point cloud within the coordinate system comprising a second plurality of spatial points, and where the second point cloud is based on the second depth map.
10. The system according to claim 9 , where the memory further includes instructions that, when executed by the at least one processor, causes the controller to:
segment the second plurality of spatial points to identify a second spatial point subset of the second point cloud and compare the second spatial point subset to the pre-identified configuration of a structure of the objects;
match the first spatial point subset in the first point cloud with the second spatial point subset in the second point cloud to orient the first point cloud with the second point cloud; and
identify motion of the objects within the field of vision of the plurality of imaging devices based on the orientation of the first point cloud relative to the second point cloud.
11. The system according to claim 10 , wherein the memory further includes instructions that, when executed by the one or more processors, causes the controller to determine a spatial trajectory of the objects based upon the identified motion of the objects from the position of the objects in the first point cloud to the position of the objects in the second point cloud.
12. The system according to claim 1 , further comprising a display device in communication with the controller,
wherein the memory further includes instructions stored thereon which, when executed by the processor, causes the controller to:
cause the display device to output an indication of a possible collision based on determining that a possible collision exists.
13. The system according to claim 12 , wherein the indication includes a three-dimensional image of a position diagram.
14. The system according to claim 12 , wherein the three-dimensional images of the position diagram illustrate a rendering of the possible collision at a later point in time in a case where the objects remain on the spatial trajectory.
15. The system according to claim 1 , wherein the memory further includes instructions stored thereon which, when executed by the processor, causes the controller to:
transmit a control signal to the robotic cart or the robotic arm to cause the robotic arm to reposition to avoid the possible collision.
16. A method for detecting potential collisions between objects of a surgical robotic system including a robotic cart supporting a robotic arm, a plurality of imaging devices supported by the robotic cart or the robotic arm, and a controller, the controller in communication with the plurality of imaging devices and the robotic arm, the method comprising:
receiving image data of a field of vision captured by the plurality of imaging devices;
generating a three-dimensional depth map including a first plurality of spatial points based on the image data; and
detecting a potential collision between objects captured in the image data based on:
the generated three-dimensional depth map;
the geometric orientation of the objects located within the field of vision; and
a sweep volume of movement of each of the objects located within the field of vision,
wherein any overlap in the sweep volumes of the objects located within the field of vision provides an indication of the potential collision.
17. The method of claim 16 , further comprising generating a first point cloud based on the three-dimensional depth map, the first point cloud comprising the plurality of spatial points.
18. The method of claim 17 , further comprising segmenting the plurality of spatial points in the point cloud to identify a spatial point subset of the point cloud, each point in the spatial point subset corresponds to a surface portion of the objects.
19. The method of claim 18 , wherein the identifying of the spatial point subset includes matching the spatial point subset to a predetermined configuration of the objects and, based on the matching, identifying the objects.
20. The method of claim 18 , further comprising:
receiving second image data from the plurality of imaging devices at a point in time later than when the image data was captured by the plurality of imaging devices;
generating a second three-dimensional depth map including a second plurality of spatial points based on the second image data;
generating a second point cloud comprising a second plurality of spatial points based on the second three-dimensional depth map;
segmenting the second plurality of spatial points to identify the spatial point subset within the second point cloud; and
matching the spatial point subset in the first point cloud with the spatial point subset in the second point cloud.
21. The method of claim 20 , further comprising:
determining a spatial trajectory of the objects based on the positional difference of the spatial point subset matched in the first point cloud and the second point cloud; and
displaying an indication of a possible collision when a possible collision exists.
22. The method according to claim 21 , further comprising:
transmitting a control signal to cause the robotic arm to modify motion in a direction to avoid the possible collision.