System and method for identification, labeling, and tracking of a medical instrument
The present invention is directed to a computing system and method for the real-time automatic identification and tracking of a medical instrument and optionally the identification of at least one anatomical object in an image generated by an imaging system. The method includes generating the image via the imaging system and providing the image to a processor of a computing system. Further, the method includes developing and training at least one machine-learned model to automatically identify or locate the medical instrument and optionally at least one machine-learned model to identify or locate the anatomical object and the surrounding tissue in the image. Moreover, the method includes automatically labeling the identified medical instrument and optionally the identified anatomical object and surrounding tissue on the image. Further, the method also includes displaying the labeled image to a user in real-time.
1. A method for labeling a medical instrument in at least one image generated by at least one imaging system, the method comprising:
obtaining, by a computing system comprising one or more computing devices, patient imaging data including the at least one image from the at least one imaging system;
inputting, by the computing system, the patient imaging data into a machine-learned medical instrument identification model;
inputting, by the computing system, the patient imaging data into a machine-learned anatomical object of interest identification model;
receiving, by the computing system as an output of the machine-learned medical instrument identification model, a first label on the at least one image corresponding with a predicted location of a tip of the medical instrument, wherein at least a portion of the medical instrument is labeled via the first label, wherein the first label identifies a tip of the medical instrument, wherein the first label comprises an outline of the tip of the medical instrument;
receiving, by the computing system as an output of the machine-learned anatomical object of interest identification model, a second label on the at least one image corresponding with a predicted location of at least a portion of an anatomical object of interest, wherein at least a portion of the anatomical object of interest is labeled via the second label, wherein the first label and the second label are overlaid onto the at least one image in real-time;
determining, by the computing system and via an objective function, a confidence level in relation to each of the predicted location of the tip of the medical instrument and the predicted location of at least a portion of the anatomical object of interest; and
displaying, by the computing system, the labeled image concurrently with an indication of at least one of the determined confidence levels.
2. The method of claim 1 , wherein the machine-learned medical instrument identification model comprises one or more of a convolutional neural network and a recurrent neural network.
3. The method of claim 1 , wherein the machine-learned anatomical object of interest identification model comprises one or more of a convolutional neural network and a recurrent neural network.
4. The method of claim 1 , wherein the first label is visually distinguishable from the second label.
5. The method of claim 1 , wherein the at least one imaging system comprises one or more of an ultrasound imaging system, a computer tomography scanner, and a magnetic resonance imaging scanner.
6. The method of claim 1 , wherein the computing system is separate from the at least one imaging system.
7. The method of claim 1 , wherein the computing system is a part of the at least one imaging system.
8. A computing system, comprising:
a machine-learned medical instrument identification model trained to label at least a portion of a medical instrument based on patient imaging data containing at least one image received from at least one imaging system;
a machine-learned anatomical object of interest identification model trained to label at least a portion of an anatomical object of interest based on the patient imaging data containing the at least one image;
one or more processors;
one or more non-transitory computer-readable media that store instructions that, when executed by the one or more processors, cause the one or more processors to perform operations, the operations comprising:
obtaining the patient imaging data containing the at least one image;
inputting the patient imaging data containing the at least one image into the machine-learned medical instrument identification model; and
receiving, as an output of the machine-learned medical instrument identification model, a first label on the image corresponding with a predicted location of a tip of the medical instrument, wherein at least a portion of the medical instrument is labeled via the first label, wherein the first label identifies a tip of the medical instrument, wherein the first label comprises an outline of the tip of the medical instrument;
inputting the patient imaging data containing the at least one image into the machine-learned anatomical object of interest identification model; and
receiving, as an output of the machine-learned anatomical object of interest identification model, a second label on the image corresponding with a predicted location of at least a portion of the anatomical object of interest, wherein at least a portion of the anatomical object of interest is labeled via the second label, wherein the first label and the second label are overlaid onto the at least one image in real-time;
determining, via an objective function, a confidence level in relation to each of the predicted location of the tip of the medical instrument and the predicted location of at least a portion of the anatomical object of interest; and
a display configured to display the labeled image to a user concurrently with an indication of at least one of the determined confidence levels.
9. The computing system of claim 8 , wherein the machine-learned medical instrument identification model comprises one or more of a convolutional neural network and a recurrent neural network.
10. The computing system of claim 8 , wherein the machine-learned anatomical object of interest identification model comprises one or more of a convolutional neural network and a recurrent neural network.
11. The computing system of claim 8 , wherein the first label is visually distinguishable from the second label.
12. The computing system of claim 8 , wherein the computing system is a part of the at least one imaging system.
13. The computing system of claim 12 , wherein the at least one imaging system comprises one or more of an ultrasound imaging system, a computer tomography scanner, and a magnetic resonance imaging scanner.