Image-based instrument identification and tracking
Disclosed is a computer-implemented method of transmitting identification information of a medical instrument. The method encompasses comparing a digital image of an instrument tray and an instrument to a digital image of just the instrument tray to determine the identity of the instrument. A characteristic geometry such as its envelope is assigned to the instrument, and a characteristic quantity of the envelope such as its aspect ratio may be used to identify the instrument. Based on determining, from the image of the instrument and the instrument tray, the relative position between those two entities, the method determines whether the instrument has been taken from the instrument tray, and accordingly instructs a medical computing system about this determination. The medical computing system may then determine whether the correct instrument has been taken from the instrument has been taken from the instrument tray, for example by comparison with medical procedure planning data.
1 . A computer-implemented method of transmitting identification information of an instrument to be used in a procedure, the method comprising:
acquiring initial state image data by a computer from an associated camera operatively coupled with the computer, the initial image data being representative of an image of an instrument tray in an initial state free of any instruments disposed thereon;
acquiring instrument state image data by the computer from the associated camera, the instrument state image data being representative of an image of the instrument tray with an instrument being disposed at an initial location on the instrument tray;
determining instrument envelope data based on the initial state image data and the instrument state image data, wherein the instrument envelope data describes a position of an envelope around the image representation of the instrument in the instrument state image data relative to a position of the instrument tray;
determining instrument position data based on the instrument envelope data, wherein the instrument position data is representative of a position of the instrument disposed at the initial location on the instrument tray;
acquiring instrument identity data based on a type of the procedure and on the position of the instrument disposed at the initial location on the instrument tray, wherein the instrument identity data identifies the instrument;
assigning the acquired instrument identity data to the instrument envelope;
determining instrument usage data based on the instrument envelope data, wherein the instrument usage data describes that the envelope around the image representation of the instrument has attained a predetermined spaced apart position relative to the position of the instrument tray; and
based on determining the instrument usage data, transmitting by the computer the acquired instrument identity data assigned to the instrument envelope data to a further associated computer of an associated computing system.
2 . The method according to claim 1 , wherein:
the acquiring the initial state image data and the instrument state image data comprises acquiring the initial state image data and the instrument state image data by the computer from the associated camera configured to image the instrument tray in the initial state free of any instruments disposed thereon and the instrument being located on the instrument tray in an infrared wavelength range.
3 . The method according to claim 1 , wherein the determining the instrument envelope data comprises determining a difference between the image of the instrument tray in the initial state free of any instruments disposed thereon and the image of the instrument being located on the instrument tray.
4 . The method according to claim 1 , further comprising tracking a position of the instrument.
5 . The method according to claim 1 , wherein the determining the instrument envelope data comprises determining instrument envelope data that describes a generic bounding shape that is independent of a physical shape of the instrument.
6 . The method according to claim 5 , wherein the determining the instrument envelope data comprises determining instrument envelope data that describes an envelope having a predetermined distance around a circumference of the image representation of the instrument.
7 . The method according to claim 1 , further comprising:
extracting from the instrument envelope data an image signature of the instrument comprising an aspect ratio of an envelope around the image representation of the instrument.
8 . The method according to claim 7 , wherein the determining the instrument identity data comprises determining the instrument identity data based on the image signature.
9 . The method according to claim 1 , further comprising acquiring infrared image data that describes an infrared image comprising an image representation of the instrument tray.
10 . The method according to claim 1 , wherein the acquiring the instrument state image data comprises acquiring an image representation of an optical code disposed on the instrument, wherein the optical code comprises information content, and wherein the instrument identity data is determined further based on the information content of the optical code.
11 . The method according to claim 1 , wherein:
the acquiring the instrument identity data comprises labelling the instrument by receiving via a user interface an instruction from the user that assigns a name or other identifier to the image representation of the instrument in the instrument state.
12 . The method according to claim 1 , wherein the acquiring the instrument identity data that identifies the instrument comprises:
determining the instrument identity data based upon:
the instrument being disposed at the initial location on the instrument tray; and
database data that relates, based upon the type of the procedure, the predetermined positions or areas of the instrument tray with types of instruments expected to be disposed at the respective positions of the instrument tray or in the respective areas of the instrument tray.
13 . The method according to claim 1 , further comprising:
determining whether a correct instrument has been taken from the instrument tray by comparison of the determined instrument identity data with medical procedure planning data of the database data corresponding to the procedure.
14 . A non-transitory computer readable storage medium storing a computer program comprising instructions that, when executed by a processor device of an associated computer, cause the associated computer to perform a method of transmitting identification information of an instrument to be used in a procedure, the method comprising:
acquiring initial state image data by a computer from an associated camera operatively coupled with the computer, the initial image data being representative of an image of an instrument tray in an initial state free of any instruments disposed thereon;
acquiring instrument state image data by the computer from the associated camera, the instrument state image data being representative of an image of the instrument tray with an instrument being disposed at an initial location on the instrument tray;
determining instrument envelope data based on the initial state image data and the instrument state image data, wherein the instrument envelope data describes a position of an envelope around the image representation of the instrument in the instrument state image data relative to a position of the instrument tray;
determining instrument position data based on the instrument envelope data, wherein the instrument position data is representative of a position of the instrument disposed at the initial location on the instrument tray;
acquiring instrument identity data based on a type of the procedure and on the position of the instrument disposed at the initial location on the instrument tray, wherein the instrument identity data that identifies the instrument;
assigning the acquired instrument identity data to the instrument envelope;
determining instrument usage data based on the instrument envelope data, wherein the instrument usage data describes that the envelope around the image representation of the instrument has attained a predetermined spaced apart position relative to the position of the instrument tray; and
based on determining the instrument usage data, transmitting by the computer the acquired instrument identity data assigned to the instrument envelope data to a further associated computer of an associated computing system.
15 . The non-transitory computer readable storage medium according to claim 14 , wherein:
the acquiring the instrument identity data comprises labelling the instrument by receiving via a user interface an instruction from the user that assigns a name or other identifier to the image representation of the instrument in the instrument state.
16 . The non-transitory computer readable storage medium according to claim 14 that, when executed by the processor device of the associated computer, causes the associated computer to:
determine the instrument identity data based upon:
the instrument being disposed at the initial location on the instrument tray; and
database data that relates, based upon the type of the procedure, the predetermined positions or areas of the instrument tray with types of instruments expected to be disposed at the respective positions of the instrument tray or in the respective areas of the instrument tray.
17 . The non-transitory computer readable storage medium according to claim 14 that, when executed by the processor device of the associated computer, causes the associated computer to:
determine whether a correct instrument has been taken from the instrument tray by comparison of the determined instrument identity data with medical procedure planning data of the database data corresponding to the procedure.
18 . A system comprising:
a computer comprising a processor, a storage medium device operatively coupled with the processor, and a computer program stored in the storage medium device; and
a camera operably coupled with the computer and being operable for generating initial state image data representative of an image of an instrument tray in an initial state free of any instruments disposed thereon and instrument state image representative of an image of an instrument to be used in a procedure and the instrument tray with the instrument being located on the instrument tray, wherein the processor is operable to execute the computer program to perform a method comprising:
acquiring the initial state image data by the camera, wherein the initial image data is representative of an image of an instrument tray in an initial state free of any instruments disposed thereon;
acquiring the instrument state image data by the camera, wherein the instrument state image data is representative of an image of the instrument tray with an instrument being disposed at an initial location on the instrument tray;
determining instrument envelope data based on the initial state image data and the instrument state image data, wherein the instrument envelope data describes a position of an envelope around the image representation of the instrument in the instrument state image data relative to a position of the instrument tray;
determining instrument position data based on the instrument envelope data, wherein the instrument position data is representative of a position of the instrument disposed at the initial location on the instrument tray;
acquiring instrument identity data based on a type of the procedure and on the position of the instrument disposed at the initial location on the instrument tray, wherein the instrument identity data identifies the instrument and
assigning the acquired instrument identity data to the instrument envelope;
determining instrument usage data based on the instrument envelope data, wherein the instrument usage data describes that the envelope around the image representation of the instrument has attained a predetermined spaced apart position relative to the position of the instrument tray; and
based on determining the instrument usage data, transmitting by the computer the acquired instrument identity data assigned to the instrument envelope data to a further associated computer of an associated computing system.
19 . The system according to claim 18 , wherein:
the acquiring the instrument identity data comprises labelling the instrument by receiving via a user interface an instruction from the user that assigns a name or other identifier to the image representation of the instrument in the instrument state.
20 . The system according to claim 18 :
wherein the acquiring the instrument identity data that identifies the instrument comprises:
determining the instrument identity data based upon:
the instrument being disposed at the initial location on the instrument tray; and
database data that relates, based upon the type of the procedure, the predetermined positions or areas of the instrument tray with types of instruments expected to be disposed at the respective positions of the instrument tray or in the respective areas of the instrument tray; and
further comprising determining whether a correct instrument has been taken from the instrument tray by comparison of the determined instrument identity data with medical procedure planning data of the database data corresponding to the procedure.