Artificial intelligence guidance system for robotic surgery
This invention is a system and method for utilizing artificial intelligence to operate a surgical robot (e.g., to perform a laminectomy), including a surgical robot, an artificial intelligence guidance system, an image recognition system, an image recognition database, and a database of past procedures with sensor data, electronic medical records, and imaging data. The image recognition system may identify the tissue type present in the patient and if it is the desired tissue type, the AI guidance system may remove a layer of that tissue with the end effector on the surgical robot, and have the surgeon define the tissue type if the image recognition system identified the tissue as anything other than the desired tissue type.
1. A computer-implemented method for generating overlay information for display upon an image captured during a surgical procedure employing a surgical robot, the method comprising:
obtaining an image during the surgical procedure, the image comprising:
a first region depicting a first tissue; and
a second region depicting a second tissue;
applying a tissue identification system to at least a portion of the image to determine:
a first tissue type identification associated with the first region;
a first position in the image associated with the first region;
a second tissue type identification associated with the second region; and
a second position in the image associated with the second region; and
causing an overlay to be generated, the overlay configured to depict:
the position of the first region in the image and the position of the second region in the image; and
an indication of a distance between the first region and the second region.
2. The computer-implemented method of claim 1 , wherein, the tissue identification system comprises a neural network, the neural network configured:
to consider the global context of the at least a portion of the image applied to the neural network; and
to determine one or more boundaries corresponding to detected tissues in the image.
3. The computer-implemented method of claim 2 , wherein the overlay is further configured to depict the first tissue type identification associated with the first region and to depict the second tissue type identification associated with the second region.
4. The computer-implemented method of claim 3 , wherein the overlay is further configured to depict:
a boundary of the first region; and
a boundary of the second region.
5. The computer-implemented method of claim 4 , the method further comprising:
determining, based, at least in part, on data from a historical image recognition database, that the distance between the first region and the second region is outside a norm; and
causing the indication of the distance in the overlay to be highlighted in response to the determination that the distance between the first region and the second region is outside the norm.
6. The computer-implemented method of claim 5 , the method further comprising:
determining that the first tissue type identification corresponds to a desired tissue type to proceed with a tissue removal, and wherein,
determining the distance between the first region and the second region is outside a norm occurs, at least in part, in response to the determining that the first tissue type identification corresponds to a desired tissue type to proceed with a tissue removal.
7. The computer-implemented method of claim 6 , wherein the image is a fluoroscopic image.
8. A non-transitory computer-readable medium comprising instructions configured to cause a computer system to perform a method for generating overlay information for display upon an image captured during a surgical procedure employing a surgical robot, the method comprising:
obtaining an image during the surgical procedure, the image comprising:
a first region depicting a first tissue; and
a second region depicting a second tissue;
applying a tissue identification system to at least a portion of the image to determine:
a first tissue type identification associated with the first region;
a first position in the image associated with the first region;
a second tissue type identification associated with the second region; and
a second position in the image associated with the second region; and
causing an overlay to be generated, the overlay configured to depict:
the position of the first region in the image and the position of the second region in the image; and
an indication of a distance between the first region and the second region.
9. The non-transitory computer-readable medium of claim 8 , wherein, the tissue identification system comprises a neural network, the neural network configured:
to consider the global context of the at least a portion of the image applied to the neural network; and
to determine one or more boundaries corresponding to detected tissues in the image.
10. The non-transitory computer-readable medium of claim 9 , wherein the overlay is further configured to depict the first tissue type identification associated with the first region and to depict the second tissue type identification associated with the second region.
11. The non-transitory computer-readable medium of claim 10 , wherein the overlay is further configured to depict:
a boundary of the first region; and
a boundary of the second region.
12. The non-transitory computer-readable medium of claim 11 , the method further comprising:
determining, based, at least in part, on data from a historical image recognition database, that the distance between the first region and the second region is outside a norm; and
causing the indication of the distance in the overlay to be highlighted in response to the determination that the distance between the first region and the second region is outside the norm.
13. The non-transitory computer-readable medium of claim 12 , the method further comprising:
determining that the first tissue type identification corresponds to a desired tissue type to proceed with a tissue removal, and wherein,
determining the distance between the first region and the second region is outside a norm occurs, at least in part, in response to the determining that the first tissue type identification corresponds to a desired tissue type to proceed with a tissue removal.
14. The non-transitory computer-readable medium of claim 13 , wherein the image is a fluoroscopic image.
15. A computer system comprising:
at least one processor; and
at least one memory, the at least one memory comprising instructions configured to cause the computer system to perform a method for generating overlay information for display upon an image captured during a surgical procedure employing a surgical robot, the method comprising:
obtaining an image during the surgical procedure, the image comprising:
a first region depicting a first tissue; and
a second region depicting a second tissue;
applying a tissue identification system to at least a portion of the image to determine:
a first tissue type identification associated with the first region;
a first position in the image associated with the first region;
a second tissue type identification associated with the second region; and
a second position in the image associated with the second region; and
causing an overlay to be generated, the overlay configured to depict:
the position of the first region in the image and the position of the second region in the image; and
an indication of a distance between the first region and the second region.
16. The computer system of claim 15 , wherein, the tissue identification system comprises a neural network, the neural network configured:
to consider the global context of the at least a portion of the image applied to the neural network; and
to determine one or more boundaries corresponding to detected tissues in the image.
17. The computer system of claim 16 , wherein the overlay is further configured to depict the first tissue type identification associated with the first region and to depict the second tissue type identification associated with the second region.
18. The computer system of claim 17 , wherein the overlay is further configured to depict:
a boundary of the first region; and
a boundary of the second region.
19. The computer system of claim 18 , the method further comprising:
determining, based, at least in part, on data from a historical image recognition database, that the distance between the first region and the second region is outside a norm; and
causing the indication of the distance in the overlay to be highlighted in response to the determination that the distance between the first region and the second region is outside the norm.
20. The computer system of claim 19 , the method further comprising:
determining that the first tissue type identification corresponds to a desired tissue type to proceed with a tissue removal, and wherein,
determining the distance between the first region and the second region is outside a norm occurs, at least in part, in response to the determining that the first tissue type identification corresponds to a desired tissue type to proceed with a tissue removal.