Object counting system using convolutional neural network for medical procedures
The present disclosure relates to a system and method for recognizing objects used in a medical procedure using a convolutional neural network. No database of image information for such objects is used or required. Rather, the neural network is trained to recognize the objects, and does not require any such image database. The system is able to reconcile the recognized objects against a ‘counted-in’ list of objects for the procedure, to ensure that all such objects are accounted for prior to closing the procedure.
1 . A system for identifying objects used in a medical procedure, comprising:
a user interface;
a camera;
at least one processor and memory configured to:
receive images of said objects after each said object is used in the medical procedure, the images being captured by the camera;
input the images into a convolutional neural network (CNN), the CNN being configured to output a classification and a confidence score for each said object in each said inputted image;
identify the objects in the images according to the outputted classifications and confidence scores therefor;
reconcile each of the objects identified in the images with a counted-in list of objects for the medical procedure; and
output information related to the reconciliation via the user interface; and
at least one of:
a bin in which the objects are disposed after being reconciled,
a continuous tape or strip to which the objects are deposited sequentially as used and prior to being imaged by said camera, and
a tray having a plurality of spaced-apart deposit stations in or onto which the objects are deposited sequentially as used and prior to being imaged by the camera.
2 . The system of claim 1 , wherein the CNN is implemented by a CNN processor and a CNN memory, of the at least one processor and memory, and is shared by the system and a plurality of other similar systems over a network.
3 . The system of claim 1 , wherein the at least one processor and memory are further configured to, for each said object:
compare the confidence score of the classification of the object to a predetermined threshold, and when the confidence score is below the predetermined threshold:
compare the classification of the object to the counted-in list of objects for the medical procedure;
adjust the confidence score of the object based on the comparison.
4 . The system of claim 3 , wherein the at least one processor and memory are further configured to:
compare the adjusted confidence score to the predetermined threshold, and when the adjusted confidence score is below the predetermined threshold:
reinput the image of the object into the CNN, the CNN being configured to output a different classification and a different confidence score of the object; and
identify the object according to the different classification and different confidence score.
5 . The system of claim 1 , wherein the at least one processor and memory are further configured to:
repeatedly retrain the CNN with the received images of the objects.
6 . The system of claim 1 , wherein the user interface is configured to output an alarm when the classification of one of the identified objects does not match an object in the counted-in list.
7 . The system of claim 1 , further comprising a backlight configured to illuminate with infrared light a field of view for capturing said images via said camera.
8 . The system of claim 7 , said infrared light being collimated light having a wavelength about 850 nm.
9 . A method comprising:
generating a counted-in list of a plurality of objects to be used during a medical procedure;
imaging the objects after use in the medical procedure, thereby generating at least one image;
inputting the at least one image into a convolutional neural network (CNN), the CNN being configured to output a classification and a confidence score for each object in the at least one image;
identifying the objects in the at least one image according to the outputted classification and confidence score;
reconciling each of the objects identified in the at least one image against the counted-in list;
outputting information related to the reconciliation;
comparing the confidence score of the classification for the object to a predetermined threshold, the confidence score being below the predetermined threshold;
thereafter comparing the classification of the object to the plurality of objects in the counted-in list; and
adjusting the confidence score of the classification for the object based on the comparison.
10 . The method of claim 9 , wherein the at least one image is input into the CNN via a network.
11 . The method of claim 9 , further comprising:
comparing the adjusted confidence score to the predetermined threshold, the adjusted score being below the predetermined threshold:
reinputting the image of the object into the CNN, the CNN being configured to output a different classification of the object and a different confidence score therefor; and
identifying the object according to the different classification and the different confidence score.
12 . The method of claim 9 , further comprising:
repeatedly retraining the CNN with the received images of the objects.
13 . The method of claim 9 ,
wherein the classification of one of the identified objects in the images does not match any object in the counted-in list, and
the method further comprises outputting an alarm indicating that the associated object has not been reconciled.
14 . The method of claim 9 , said objects being imaged by a camera along a field of view that is backlit with collimated light at a wavelength of about 850 nm.
15 . The method of claim 9 , said at least one image being preprocessed by applying a static color threshold to segment a said object therein from background in the at least one image so that only that portion of said at least one image comprising that object is input into the CNN for classification of that object.
16 . A system comprising:
at least one processor and memory configured to implement a convolutional neural network (CNN), wherein the CNN is configured to:
receive images of objects used during a medical procedure;
output a classification of each object in the images and a confidence score corresponding to each said classification;
re-receive an image of a said object used during the medical procedure for which the CNN had outputted a prior classification and prior confidence score below a pre-determined threshold; and
receive the prior classification and the prior confidence score of the object; and
output a different classification of the object in the re-received image, and a different confidence score therefor, the different classification of the object and the different confidence score being at least in part based on the received prior classification and the received prior confidence score,
wherein the CNN is configured to receive the images via a network from a plurality of remote locations.