Guiding medically invasive devices with radiation absorbing markers via image processing
A system and method is disclosed for guiding an invasive medical device with radiation absorbing markers. The invasive medical device can include markers with different radiation absorbing properties relative to other portions of the invasive medical device. An imaging device can generate images of the invasive medical device within a patient. A trained model for the invasive medical device can be trained on annotated images of the invasive medical device annotated with marker information identifying the markers and spatial information for the invasive medical device. An imaging computer system can apply the trained model to images of the invasive medical device within the patient including depictions of the markers to determine current spatial information of the invasive medical device inside the patient. The images of the invasive medical device and visual spatial information representing the spatial information of the invasive medical device can be outputted to a display.
1. A system for augmenting imaging data depicting an invasive medical device, the system comprising:
an invasive medical device configured to be inserted into a patient as part of a medical procedure, wherein the invasive medical device includes one or more markers with different radiation absorbing properties relative to other portions of the invasive medical device;
an imaging device configured to generate one or more images of the invasive medical device inside the patient, wherein the imaging device is separate from the invasive medical device and is capable of being positioned at a vantage point relative to the patient;
a database programmed to store a trained model for the invasive medical device, the trained model being generated from one or more machine learning algorithms trained on annotated images of the invasive medical device annotated with (i) marker information identifying the one or more markers and (ii) spatial information for the invasive medical device, wherein the trained model is used to determine spatial information for the invasive medical device within the patient from unannotated images of the invasive medical device;
an imaging computer system programmed to:
receive the one or more images of the invasive medical device inside the patient from the imaging device, wherein the one or more images include depictions of the one or more markers,
access the trained model for the invasive medical device from the database,
determine current spatial information of the invasive medical device inside the patient by applying the trained model to the one or more images of the invasive medical device with the depictions of the one or more markers, and
output the current spatial information of the invasive medical device; and
a display configured to monitor the invasive medical device inside the patient, wherein the display is programmed to output (i) the one or more images of the invasive medical device inside the patient as captured by the imaging device, and (ii) visual spatial information representing the current spatial information of the invasive medical device inside the patient as determined from application of the trained model to the one or more images.
2. The system of claim 1 , further comprising:
a training computer system to generate the trained model for the invasive medical device having the one or more markers, the training computer system being programmed to:
obtain the annotated images of the invasive medical device having the one or more markers with spatial information, the annotated images depicting the invasive medical device inside a patient, the spatial information identifying spatial postures of the invasive medical device having the one or more markers in the annotated images,
iteratively train a model for the invasive medical device by correlating each of the annotated images to corresponding spatial information across one or more model layers using the one or more machine learning algorithms, wherein the iterative training generates the trained model for the invasive medical device, and
store the trained model for use by the imaging computer system.
3. The system of claim 2 , wherein at least a portion of the annotated images are actual images from use of the invasive medical device having the one or more markers that have been manually annotated by a practitioner with spatial information.
4. The system of claim 2 , wherein at least a portion of the annotated images are computer generated images simulating use of the invasive medical device inside patients as imaged by the imaging device.
5. The system of claim 2 , wherein the one or more machine learning algorithms comprises a supervised deep learning algorithm.
6. The system of claim 2 , wherein the trained model comprises a long short-term memory model.
7. The system of claim 1 , wherein the one or more markers comprise one or more alterations to the invasive medical device that have a non-medical application.
8. The system of claim 1 , wherein:
the trained model is specific to the combination of the invasive medical device having the portion coated with the radiation absorption material and the imaging device, and
other trained models are used for other combinations of (i) the invasive medical device or other invasive medical devices and (ii) the imaging device or other imaging devices.
9. The system of claim 1 , wherein the imaging device comprises an x-ray imaging device and the one or more images comprises one or more x-ray images.
10. The system of claim 1 , wherein the imaging device comprises an ultrasound device and the one or more images comprises one or more ultrasound images.
11. The system of claim 1 , wherein the imaging device comprises a computerized tomography (“CT”) imaging device and the one or more images comprises one or more CT scans.
12. The system of claim 1 , wherein the imaging device comprises a magnetic resonance imaging (“MRI”) device and the one or more images comprises one or more MRI images.
13. The system of claim 1 , wherein:
the spatial information in the annotated images comprises orientation information that identifies orientations of the invasive medical device in the annotated images,
the current spatial information comprises a current orientation of the invasive medical device within the patient, and
the visual spatial information comprises visual orientation information representing the current orientation of the invasive medical device within the patient as determined from application of the trained model to the one or more images.
14. The system of claim 13 , wherein:
the current orientation comprises a current roll, pitch, and yaw of the invasive medical device, and
the visual orientation information includes visual roll information, visual pitch information, and visual yaw information that is presented in the display with the one or more images and the visual position information.
15. The system of claim 13 , wherein the current roll, pitch, and yaw is discretized and the trained model is determined using a sigmoid-cross entropy.
16. The system of claim 1 , wherein:
the spatial information in the annotated images comprises position information that identifies positions of the invasive medical device in the annotated images,
the current spatial information comprises a current position of the invasive medical device within the patient, and
the visual spatial information comprises visual position information representing the current position of the invasive medical device within the patient as determined from application of the trained model to the one or more images.
17. The system of claim 16 , wherein:
the current position comprises (i) anterior and posterior position information, (ii) cranial and caudal position information, and (iii) left and right position information, and
the visual position information includes visual anterior and posterior position information, visual cranial and caudal position information, and visual left and right position information that is presented in the display with the one or more images and the visual orientation information.
18. The system of claim 1 , wherein:
the spatial information in the annotated images comprises orientation and position information that identifies orientations and positions of the invasive medical device in the annotated images,
the current spatial information comprises a current orientation and a current position of the invasive medical device within the patient, and
the visual spatial information comprises visual orientation information representing the current orientation of the invasive medical device within the patient and visual position information representing the current position of the invasive medical device within the patient.
19. The system of claim 1 , wherein the one or more markers comprise one or more portions of the invasive medical device being coated with a material that absorbs emitted photons differently from the other portions of the invasive medical device that are either uncoated or coated in a different material.
20. The system of claim 1 , wherein the one or more markers comprise one or more portions of the invasive medical device being made of a material that absorbs emitted photons differently from the other portions of the invasive medical device made of a different material.