Presurgical planning
A system, method, and computer program product for presurgical planning related to an object associated with a medical procedure for a patient is disclosed. A model can be trained for an object as part of a medical procedure, the trained model being generated from one or more machine learning algorithms that are trained on annotated images of the object with spatial information of the object. An imaging computer system can receive or simulate one or more images of the object captured by an imaging device positioned external to the patient. The imaging computer system can further determine, based on applying the trained model to the one or more images of the object, spatial information of the object during the medical procedure. A display can output the one or more images and the spatial information of the object.
1 . A system for presurgical planning related to an object internal to a patient, the system comprising:
a database configured to store a trained model, the trained model being generated from one or more machine learning algorithms that are trained on training images of internal patient objects with annotated information, wherein the trained model is configured to be used to infer information for presurgical planning;
a computer system that is configured to:
receive one or more images of an object internal to a patient captured by an imaging device, wherein the imaging device is configured to capture the one or more images of the object from a position external to the patient,
access the trained model from the database,
determine, based on applying the trained model to the one or more images of the object, inference information for presurgical planning related to the object internal to the patient, and
provide the inference information for presurgical planning related to the object internal to the patient; and
a display to provide presurgical planning for the patient, the display being configured to output at least the inference information related to the object internal to the patient as determined based on application of the trained model to the one or more images.
2 . The system of claim 1 , further comprising:
a training computer system to generate the trained model, the training computer system being programmed to:
obtain the training images of the internal patient objects with the annotated information, the training images depicting the internal patient objects, the annotated information identifying features of the internal patient objects for use in presurgical planning,
iteratively train a model for the internal patient objects by correlating each of the training images to corresponding annotated information across one or more model layers using the one or more machine learning algorithms, wherein the iterative training generates the trained model for the internal patient objects, and
store the trained model for use by the computer system.
3 . The system of claim 2 , wherein the one or more machine learning algorithms comprises a supervised deep learning algorithm.
4 . The system of claim 2 , wherein the trained model comprises a long short-term memory model.
5 . The system of claim 1 , wherein:
the computer system is further configured render one or more simulated images of the object, and
the display is configured to output the inference information based, at least in part, on the one or more simulated images of the object internal to the patient.
6 . The system of claim 1 , wherein the object comprises an invasive medical device.
7 . The system of claim 6 , wherein the invasive medical device comprises a needle, a wire, a catheter, a probe, a stent, a drain, or a snare.
8 . The system of claim 1 , wherein:
the system is further configured to monitor a plurality of objects internal to the patient,
the computer system is further configured to determine the inference information for the plurality of objects, and
the display is further configured to output the inference information for the plurality of objects.
9 . The system of claim 8 , wherein the plurality of objects include at least (i) a first object comprising an invasive medical device and (ii) a second object comprising an anatomical structure of the patient.
10 . The system of claim 9 , wherein the inference information for the plurality of objects comprises spatial information for the first object and the second object relative to each other.
11 . The system of claim 8 , wherein at least one of the plurality of objects comprises an anatomical structure of the patient.
12 . The system of claim 1 , wherein the object comprises an anatomical structure of the patient.
13 . The system of claim 1 , wherein the one or more images comprise one or more x-ray images, one or more ultrasound images, one or more computerized tomography (“CT”) scans, one or more magnetic resonance imaging (“MRI”) images, or one or more fluoroscopic images.
14 . The system of claim 1 , wherein the inference information comprises spatial information for the object.
15 . The system of claim 14 , wherein the spatial information comprises orientation information that specifies an orientation of at least a portion of the object internal to the patient.
16 . The system of claim 14 , wherein the spatial information comprises position information that specifies a position of at least a portion of the object internal to the patient.
17 . The system of claim 1 , wherein the inference information comprises identifying information for the object that identifies at least a portion of the object internal to the patient.
18 . A method for presurgical planning related to an object internal to a patient, the method comprising:
receiving one or more images of an object internal to a patient captured by an imaging device, wherein the imaging device is configured to capture the one or more images of the object from a position external to the patient;
accessing a trained model from a database, wherein the trained model is stored in the database and generated from one or more machine learning algorithms trained on training images of internal patient objects with annotated information, wherein the trained model is configured to be used to determine information for presurgical planning;
determining, based on applying the trained model to the one or more images of the object, inference information for presurgical planning related to the object internal to the patient; and
providing the inference information for presurgical planning related to the object for presentation on a display, the display being configured to output at least the inference information related to the object internal to the patient as determined based on application of the trained model to the one or more images.
19 . The method of claim 18 , wherein the object comprises an invasive medical device.
20 . A non-transitory computer readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:
receiving one or more images of an object internal to a patient captured by an imaging device, wherein the imaging device is configured to capture the one or more images of the object from a position external to the patient;
accessing a trained model for from a database, wherein the trained model is stored in the database and generated from one or more machine learning algorithms trained on training images of internal patient objects with annotated information, wherein the trained model is configured to be used to determine information for presurgical planning;
determining, based on applying the trained model to the one or more images of the object, inference information for presurgical planning related to the object internal to the patient; and
providing the inference information for presurgical planning related to the object for presentation on a display, the display being configured to output at least the inference information related to the object internal to the patient as determined based on application of the trained model to the one or more images.
21 . The system of claim 2 , wherein the one or more model layers comprise a convolutional layer, one or more pooling layers, and a plurality of residual layers.
22 . The system of claim 21 , wherein applying the trained model to the one or more images of the object comprises, for each of the one or more images, feeding the image sequentially though (i) the convolutional layer, (ii) a first of the one or more pooling layer, (iii) each of the plurality of residual layers, and (iv) a second of the one or more pooling layers.
23 . The system of claim 22 , wherein each of the plurality of residual layers comprises a subnetwork with one or more pathways of convolutional layers and a sum layer combining the outputs of the one or more pathways.
24 . The system of claim 21 , wherein the convolutional layer comprises a K×K convolution with N filters to be applied at each input location to the convolution layer at one or more specified strides.
25 . The method of claim 18 , wherein the object comprises an anatomical structure of the patient.
26 . The method of claim 18 , wherein the one or more machine learning algorithms comprises a supervised deep learning algorithm and the trained model comprises a long short-term memory model.
27 . The method of claim 18 , further comprising:
generating the trained model by performing operations comprising:
obtaining the training images of the internal patient objects with the annotated information, the training images depicting the internal patient objects, the annotated information identifying features of the internal patient objects for use in presurgical planning,
iteratively training a model for the internal patient objects by correlating each of the training images to corresponding annotated information across one or more model layers using the one or more machine learning algorithms, wherein the iterative training generates the trained model for the internal patient objects, and
storing the trained model for use by the computer system.
28 . The method of claim 27 , wherein:
the one or more model layers comprise a convolutional layer, one or more pooling layers, and a plurality of residual layers, and
applying the trained model to the one or more images of the object comprises, for each of the one or more images, feeding the image sequentially though (i) the convolutional layer, (ii) a first of the one or more pooling layer, (iii) each of the plurality of residual layers, and (iv) a second of the one or more pooling layers.
29 . The non-transitory computer readable medium of claim 20 , wherein the object comprises an invasive medical device.
30 . The non-transitory computer readable medium of claim 20 , wherein the object comprises an anatomical structure of the patient.