Colorization of medical devices in robotic surgery using AI and machine learning

1 . A computer-implemented method for intraoperative medical instrument recognition, the computer-implemented method comprising:

receiving, from an imaging device positioned inside of a patient, a stream of intraoperative three-dimensional (3D) imaging data that includes anatomy of a patient and at least one medical instrument;

applying a material recognition algorithm to identify one or more objects formed of a predetermined material present in the intraoperative 3D imaging data, wherein the material recognition algorithm includes execution of a first machine-learning model that has been trained, based on training images of medical instruments in anatomy of one or more individuals and training material labels applied to the medical instruments, to predict a likelihood that a portion or region of input imaging data includes a particular material and to segment or generate a reconstruction of a shape formed by the particular material;

applying a shape recognition algorithm to the one or more identified objects to identify the at least one medical instrument, wherein the shape recognition algorithm includes execution of a second machine-learning model that has been trained, based on training shapes and medical instrument labels applied to the training shapes, to predict a likelihood that an input shape corresponds to a particular medical instrument; and

generating a modified intraoperative imaging data stream in real-time with the receiving of the stream of intraoperative 3D imaging data, wherein the modified intraoperative imaging data stream includes a visual characteristic applied to a region of the intraoperative 3D imaging data corresponding to the at least one identified medical instrument.

2 . The computer-implemented method of claim 1 , further comprising:

causing a display device to output the modified intraoperative imaging data stream in real-time with the receiving of the stream of intraoperative 3D imaging data.

3 . The computer-implemented method of claim 1 , further comprising:

transmitting the modified intraoperative imaging data stream to a robotic surgery device configured to manipulate the at least one medical instrument based on one or more of a position, orientation, or motion of the at least one medical instrument indicated by the visual characteristic applied to the intraoperative 3D imaging data.

4 . The computer-implemented method of claim 1 , wherein the visual characteristic is indicative of one or more of orientation position or motion of the at least one medical instrument.

5 . The computer-implemented method of claim 1 , wherein:

the at least one medical instrument is a plurality of medical instruments; and

a unique visual characteristic is used to identify each medical instrument.

6 . The computer-implemented method of claim 1 , wherein the visual characteristic includes a colorization.

7 . The computer-implemented method of claim 1 , wherein the intraoperative 3D imaging data includes one or more of 3D ultrasound imaging data or 3D video.

8 . The computer-implemented method of claim 1 , wherein:

the generating of the modified intraoperative imaging data stream is based on the stream of intraoperative 3D imaging data over a period of time, such that the generating includes predicting one or more of a future position, orientation, or motion of the at least one medical instrument or a position, orientation, or motion of an occluded portion of the at least one medical instrument; and

the region where the visual characteristic is applied is based on the predicting.

9 . The computer-implemented method of claim 1 , wherein the training shapes used to train the second machine-learning model were generated by the first machine-learning model.

10 . A computer-implemented method for object recognition during a medical procedure, comprising:

receiving, from an imaging device positioned inside of a patient, an imaging scene of a target, the imaging scene including a medical instrument;

applying a metal recognition algorithm to identify metal present in the imaging scene, wherein the metal recognition algorithm includes execution of a first machine-learning model that has been trained, based on training images of medical instruments in anatomy of one or more individuals and training metal labels applied to the medical instruments, to predict a likelihood that a portion or region of an input imaging scene includes a metal and to segment or generate a reconstruction of a shape formed by the metal;

applying an object recognition algorithm to the identified metal to identify the medical instrument, wherein the object recognition algorithm includes execution of a second machine-learning model that has been trained, based on training shapes and medical instrument labels applied to the training shapes, to predict a likelihood that an input shape corresponds to a particular medical instrument; and

generating a modified imaging scene that includes a visual characteristic applied to the identified medical instrument.

11 . The computer-implemented method of claim 10 , further comprising:

causing a display device to output the modified imaging scene in real-time with the receiving of the imaging scene from the imaging device.

12 . The computer-implemented method of claim 10 , further comprising:

transmitting the modified imaging scene to a robotic surgery device configured to manipulate the medical instrument based on one or more of a position, orientation, or motion of the medical instrument indicated by the visual characteristic applied to the imaging scene.

13 . The computer-implemented method of claim 10 , wherein the visual characteristic is indicative of one or more of orientation, position, or motion of the medical instrument.

14 . The computer-implemented method of claim 10 , wherein the visual characteristic includes a colorization.

15 . The computer-implemented method of claim 10 , wherein the imaging device includes one or more of a 3D ultrasound probe or a 3D video camera.

16 . The computer-implemented method of claim 10 , wherein:

the generating of the modified imaging scene is based on data from the imaging scene taken over a period of time, such that the generating includes predicting one or more of a future position, orientation, or motion of the medical instrument or a position, orientation, or motion of an occluded portion of the medical instrument; and

a location in the imaging scene where the visual characteristic is applied is based on the predicting.

17 . The computer-implemented method of claim 10 , wherein the training shapes used to train the second machine-learning model were generated by the first machine-learning model.

18 . A system for intraoperative medical instrument recognition, comprising:

at least one medical instrument;

at least one imaging device configured to capture intraoperative three-dimensional (3D) imaging data; and

an imaging analysis device that includes:

at least one memory storing:

instructions for intraoperative medical instrument recognition;

a first machine-learning model that has been trained, based on training images of medical instruments in anatomy of one or more individuals and training metal labels applied to the medical instruments, to predict a likelihood that a portion or region of an input imaging scene includes a metal and to segment or generate a reconstruction of a shape formed by the metal; and

a second machine-learning device that has been trained, based on training shapes and medical instrument labels applied to the training shapes, to predict a likelihood that an input shape corresponds to a particular medical instrument; and

at least one processor operatively connected to the at least one memory and configured to execute the instructions to perform operations including:

receiving, from the imaging device, a stream of intraoperative 3D imaging data that includes anatomy of a patient and at least one medical instrument at least partially inserted into the anatomy;

applying the first machine-learning model to the intraoperative 3D imaging data to identify one or more regions of the intraoperative 3D imaging data that include the metal, and to segment or generate a shape of the metal;

applying the second machine-learning model to the shape to identify the at least one medical instrument; and

generating a modified intraoperative imaging data stream in real-time with the receiving of the stream of intraoperative 3D imaging data, wherein the modified intraoperative imaging data stream includes a visual characteristic applied to a region of the intraoperative 3D imaging data corresponding to the at least one identified medical instrument.