Robotic surgery depth detection and modeling

1. A computer-implemented method, comprising:

receiving procedure type data describing a surgical procedure to be performed;

receiving first image data corresponding to preoperative imaging of a surgical location, the surgical location based on the procedure type data;

receiving second image data from a camera connected to a robotic arm of a robotic surgical system;

receiving kinematic data describing a position and orientation of the robotic arm in relation to the surgical location;

generating a feature model of the surgical location based on the image data and the kinematic data;

generating an image-mapped feature model by mapping the first image data onto the feature model;

determining a sequence of procedure steps based on the procedure type data and the image-mapped feature model;

determining a current status of the surgical procedure based on the image-mapped feature model and current kinematic data using a pattern recognition algorithm, wherein the image-mapped feature model represents a first portion of anatomy yet to be removed and a second portion of anatomy that has already been removed; and

providing, for display at a display device, (i) the sequence of procedure steps, and (ii) the current status of the surgical procedure, using the image-mapped feature model, by showing a first representation of the first portion of anatomy yet to be removed and a second representation of the second portion of anatomy that has already been removed, wherein the second representation of the second portion of anatomy that has already been removed comprises a shadow portion of the image-mapped feature model.

2. The computer-implemented method of claim 1 , wherein determining the sequence of procedure steps comprises:

accessing a database of procedure steps; and

selecting the sequence of procedure steps based on the procedure type data.

3. The computer-implemented method of claim 1 , further comprising:

determining, based on the second image data, a current procedure step of the sequence of procedure steps corresponding to a present stage of the procedure; and

providing, for display at the display device, the current procedure step.

4. The computer-implemented method of claim 1 , further comprising generating an augmented reality display including the image-mapped feature model and the sequence of procedure steps.

5. The computer-implemented method of claim 1 , wherein mapping the first image data onto the feature model comprises:

determining a first number of distinct points in the first image data;

determining a second number of distinct points in the feature model; and

mapping the first image data onto the feature model by correlating the first number of distinct points with the second number of distinct points.

6. The computer-implemented method of claim 1 , wherein the second image data comprises distance data from a depth sensing device connected to the camera and corresponding to distances between the camera and features of the surgical location.

7. The computer-implemented method of claim 1 , wherein the first image data comprises at least one of:

magnetic resonance image data;

radiography;

ultrasound;

magnetic particle imaging;

computed tomography image data; or

positron emission tomography image data.

8. The computer-implemented method of claim 1 , wherein the second image data represents anatomy at the surgical location, the method further comprising, following a first procedure step of the sequence of procedure steps:

acquiring third image data from the camera, the third image data representing a change in the anatomy at the surgical location;

updating the feature model of the surgical location based on the third image data;

generating an updated image-mapped feature model by mapping the third image data onto the feature model; and

providing, for display at the display device, the updated image-mapped feature model.

9. The computer-implemented method of claim 1 , further comprising determining the kinematic data using a kinematic model of the robotic arm.

10. The computer-implemented method of claim 1 , wherein the robotic arm of the robotic surgical system is controlled by a robotic surgical console.

11. A robotic surgical system comprising:

a plurality of robotic arms to perform a surgical procedure;

a camera positioned on a first robotic arm of the plurality of robotic arms;

a display device to display output from the camera;

one or more processors; and

one or more non-transitory computer-readable media comprising processor-executable instructions that, when executed by the one or more processors, cause the one or more processors to:

receive procedure type data describing the surgical procedure to be performed by the robotic surgical system;

receive first image data corresponding to preoperative imaging of a surgical location, the surgical location based on the procedure type data;

receive second image data from the camera including a present field of view of the camera;

receive kinematic data describing a position of an end of the first robotic arm in relation to the surgical location;

generate a feature model of the surgical location based on the second image data and the kinematic data;

generate an image-mapped feature model by mapping the first image data onto the feature model;

determine a sequence of procedure steps based on the procedure type data and the image-mapped feature model; and

determining a current status of the surgical procedure based on the image-mapped feature model and current kinematic data using a pattern recognition algorithm, wherein the image-mapped feature model represents a first portion of anatomy yet to be removed and a second portion of anatomy that has already been removed; and

provide, for display at the display device, (i) the sequence of procedure steps, and (ii) the current status of the surgical procedure, using the image-mapped feature model, by showing a first representation of the first portion of anatomy yet to be removed and a second representation of the second portion of anatomy that has already been removed, wherein the second representation of the second portion of anatomy that has already been removed comprises a transparent portion of the image-mapped feature model.

12. The robotic surgical system of claim 11 , wherein the camera is a stereoscopic endoscope.

13. The robotic surgical system of claim 11 , wherein the camera comprises a depth sensing device.

14. The robotic surgical system of claim 13 , wherein the depth sensing device is at least one of:

a second camera;

an ultrasound sensor;

a radio detection and ranging sensor; or

a light detection and ranging sensor.

15. The robotic surgical system of claim 11 , wherein mapping the first image data onto the feature model comprises performing iterative closest-point registration.

16. The robotic surgical system of claim 11 , wherein determining the sequence of procedure steps comprises:

accessing a database of procedure steps; and

selecting the sequence of procedure steps based on the procedure type data.

17. The robotic surgical system of claim 11 , wherein the one or more non-transitory computer-readable media comprising the processor-executable instructions that, when executed by the one or more processors, further cause the one or more processors to:

determining, based on the second image data, a current procedure step of the sequence of procedure steps corresponding to a present stage of the procedure; and

providing, for display at the display device, the current procedure step.