System for monitoring object inserted into patient's body via image processing
A system and method for image-based monitoring of an object inserted into a patient is disclosed. A model can be trained for an object configured to be inserted into a patient as part of a medical procedure, the trained model being generated from one or more machine learning algorithms that are trained on annotated images of the object with spatial information of the object. An imaging computer system can receive one or more images of the object inserted within the patient captured by an imaging device positioned external to the patient. The imaging computer system can further determine, based on applying the trained model to the one or more images of the object, current spatial information of the object within the patient. A display can output the one or more images and the current spatial information of the object.
1. A system for image-based monitoring of an object inserted into a patient, the system comprising:
a database configured to store a trained model for an object configured to be inserted into a patient as part of a medical procedure, the trained model being generated from one or more machine learning algorithms that are trained on annotated images of the object with spatial information of the object, wherein the trained model is configured to be used to determine spatial information from unannotated images of the object;
an imaging computer system that is configured to:
receive, from an imaging device, one or more images of the object inserted within the patient, wherein the imaging device is configured to capture the one or more images of the object from a position external to the patient,
access the trained model for the object from the database,
determine, based on applying the trained model to the one or more images of the object, current spatial information of the object within the patient, and
provide the current spatial information of the object within the patient; and
a display to monitor the object within the patient, the display being configured to output (i) the one or more images of the object as captured by the imaging device from the position external to the patient, and (ii) the current spatial information of the object within the patient as determined based on application of the trained model to the one or more images.
2. The system of claim 1 , further comprising:
a training computer system to generate the trained model for the object, the training computer system being programmed to:
obtain the annotated images of the object with the spatial information, the annotated images depicting the object within a patient, the spatial information identifying spatial postures of the object within the patient,
iteratively train a model for the object by correlating each of the annotated images to corresponding spatial information across one or more model layers using the one or more machine learning algorithms, wherein the iterative training generates the trained model for the object, and
store the trained model for use by the imaging computer system.
3. The system of claim 2 , wherein at least a portion of the annotated images are actual images from use of the object that have been manually annotated by a practitioner with spatial information.
4. The system of claim 2 , wherein at least a portion of the annotated images are computer generated images simulating the object within patients as imaged by the imaging device.
5. The system of claim 2 , wherein the one or more machine learning algorithms comprises a supervised deep learning algorithm.
6. The system of claim 2 , wherein the trained model comprises a long short-term memory model.
7. The system of claim 1 , wherein the object comprises a stent, a drain, or a snare.
8. The system of claim 1 , wherein the object comprises a needle, a wire, a catheter, or a probe.
9. The system of claim 1 , wherein:
the system is further configured to monitor a plurality of objects inserted into the patient,
the imaging computer system is further configured to determine the current spatial information for the plurality of objects, and
the display is further configured to output the current spatial information for the plurality of objects.
10. The system of claim 1 , wherein the imaging device comprises an x-ray imaging device and the one or more images comprises one or more x-ray images.
11. The system of claim 1 , wherein the imaging device comprises an ultrasound device and the one or more images comprises one or more ultrasound images.
12. The system of claim 1 , wherein the imaging device comprises a computerized tomography (“CT”) imaging device and the one or more images comprises one or more CT scans.
13. The system of claim 1 , wherein the imaging device comprises a magnetic resonance imaging (“MRI”) device and the one or more images comprises one or more MRI images.
14. The system of claim 1 , wherein the imaging device comprises a fluoroscope and the one or more images comprises one or more fluoroscopic images.
15. The system of claim 1 , wherein:
the spatial information in the annotated images comprises orientation information that identifies orientations of the object in the annotated images,
the current spatial information comprises a current orientation of the object within the patient.
16. The system of claim 1 , wherein:
the spatial information in the annotated images comprises position information that identifies positions of the object in the annotated images,
the current spatial information comprises a current position of the object within the patient.
17. The system of claim 1 , wherein:
the spatial information in the annotated images comprises orientation and position information that identifies orientations and positions of the object in the annotated images,
the current spatial information comprises a current orientation and a current position of the object within the patient.
18. The system of claim 1 , wherein the current spatial information is determined relative to one or more anatomical structures within the patient.
19. The system of claim 1 , wherein the current spatial information is determined relative to one or more other objects inserted within the patient.
20. The system of claim 1 , wherein the current spatial information is determined with regard to a fixed or predefined origin and canonical orientation.
21. A method for image-based monitoring of an object inserted into a patient, the method comprising:
receiving, from an imaging device, one or more images of the object inserted within the patient, wherein the imaging device is configured to capture the one or more images of the object from a position external to the patient;
accessing a trained model for the object from a database, wherein the trained model is stored in the database and generated from one or more machine learning algorithms trained on annotated images with spatial information of the object, wherein the trained model is configured to be used to determine spatial information from unannotated images of the object;
determining, based on applying the trained model to the one or more images of the object, current spatial information of the object within the patient; and
providing the current spatial information of the object for presentation on a display, the display being configured to output (i) the one or more images of the object as captured by the imaging device from the position external to the patient, and (ii) the current spatial information of the object within the patient as determined based on application of the trained model to the one or more images.
22. The method of claim 21 , further comprising:
generating the trained model for the object, wherein generating the trained model comprises:
obtaining the annotated images of the object with the spatial information, the annotated images depicting the object within a patient, the spatial information identifying spatial postures of the object within the patient,
iteratively training a model for the object by correlating each of the annotated images to corresponding spatial information across one or more model layers using the one or more machine learning algorithms, wherein the iterative training generates the trained model for the object, and
storing the trained model in the database.
23. The method of claim 22 , wherein at least a portion of the annotated images are actual images from use of the object that have been manually annotated by a practitioner with spatial information.
24. The method of claim 22 , wherein at least a portion of the annotated images are computer generated images simulating the object within patients as imaged by the imaging device.
25. The method of claim 22 , wherein the one or more machine learning algorithms comprises a supervised deep learning algorithm.
26. The method of claim 22 , wherein the trained model comprises a long short-term memory model.
27. The method of claim 21 , wherein the object comprises a stent, a drain, or a snare.
28. The method of claim 21 , wherein the object comprises a needle, a wire, a catheter, or a probe.
29. The method of claim 21 , wherein:
the object comprises a plurality of objects inserted into the patient,
the current spatial information is determined for the plurality of objects, and
the display is further configured to output the current spatial information for the plurality of objects.
30. The method of claim 21 , wherein the imaging device comprises an x-ray imaging device and the one or more images comprises one or more x-ray images.
31. The method of claim 21 , wherein the imaging device comprises an ultrasound device and the one or more images comprises one or more ultrasound images.
32. The method of claim 21 , wherein the imaging device comprises a computerized tomography (“CT”) imaging device and the one or more images comprises one or more CT scans.
33. The method of claim 21 , wherein the imaging device comprises a magnetic resonance imaging (“MRI”) device and the one or more images comprises one or more MRI images.
34. The method of claim 21 , wherein the imaging device comprises a fluoroscope and the one or more images comprises one or more fluoroscopic images.
35. The method of claim 21 , wherein:
the spatial information in the annotated images comprises orientation information that identifies orientations of the object in the annotated images,
the current spatial information comprises a current orientation of the object within the patient.
36. The method of claim 21 , wherein:
the spatial information in the annotated images comprises position information that identifies positions of the object in the annotated images,
the current spatial information comprises a current position of the object within the patient.
37. The method of claim 21 , wherein:
the spatial information in the annotated images comprises orientation and position information that identifies orientations and positions of the object in the annotated images,
the current spatial information comprises a current orientation and a current position of the object within the patient.
38. The method of claim 21 , wherein the current spatial information is determined relative to one or more anatomical structures within the patient.
39. The method of claim 21 , wherein the current spatial information is determined relative to one or more other objects inserted within the patient.
40. The method of claim 21 , wherein the current spatial information is determined with regard to a fixed or predefined origin and canonical orientation.
41. A non-transitory computer readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising: receiving, from an imaging device, one or more images of the object inserted within the patient, wherein the imaging device is configured to capture the one or more images of the object from a position external to the patient; accessing a trained model for the object from a database, wherein the trained model is stored in the database and generated from one or more machine learning algorithms trained on annotated images with spatial information of the object, wherein the trained model is configured to be used to determine spatial information from unannotated images of the object; determining, based on applying the trained model to the one or more images of the object, current spatial information of the object within the patient; and providing the current spatial information of the object for presentation on a display, the display being configured to output (i) the one or more images of the object as captured by the imaging device from the position external to the patient, and (ii) the current spatial information of the object within the patient as determined based on application of the trained model to the one or more images.
42. The computer readable medium of claim 41 , wherein:
the spatial information in the annotated images comprises orientation information that identifies orientations of the object in the annotated images,
the current spatial information comprises a current orientation of the object within the patient.
43. The computer readable medium of claim 41 , wherein:
the spatial information in the annotated images comprises position information that identifies positions of the object in the annotated images,
the current spatial information comprises a current position of the object within the patient.
44. The computer readable medium of claim 41 , wherein:
the spatial information in the annotated images comprises orientation and position information that identifies orientations and positions of the object in the annotated images,
the current spatial information comprises a current orientation and a current position of the object within the patient.
45. The computer readable medium of claim 41 , wherein the current spatial information is determined relative to one or more anatomical structures within the patient.
46. The computer readable medium of claim 41 , wherein the current spatial information is determined relative to one or more other objects inserted within the patient.
47. The computer readable medium of claim 41 , wherein the current spatial information is determined with regard to a fixed or predefined origin and canonical orientation.