Granted Patent B2

US 12,611,086 · App. 17/790,806 · Granted Apr 28, 2026

Inventors: Dwight Meglan (Westwood, MA); Eric J. Taylor (Southington, CT)

Assignee: Covidien LP

A61B1/000094A61B1/046A61B1/0638

Abstract

Claims (63)

1 . A system for object identification in endoscopy images, comprising:

a light source configured to provide light within a surgical operative site, the light source configured to simultaneously or alternatively produce a first light including an infrared (IR) band and a second light including a visible band;

an imaging device configured to acquire images from the surgical operative site;

an imaging device control unit configured to control the imaging device, the control unit including:

a processor; and

a memory storing instructions thereon, which, when executed by the processor, cause the system to:

provide, from the light source, light within the surgical operative site including at least one of the first light or the second light;

switch the light source to produce only the first light;

capture an image of an object located in the surgical operative site, by the imaging device, the image including only the first light from the object, the object including an infrared (IR) marking applied to the object, the IR marking including data corresponding to information about a type of the object;

access the image;

perform real-time image recognition on the image to detect the IR marking;

perform real-time image recognition on the image to detect the object and classify the object, based on the IR marking applied to the object;

generate an augmented image where the IR marking is removed from the image; and

display the augmented image on a display.

2 . The system of claim 1 , wherein performing real-time image recognition, based on the IR marking, includes amplifying the IR marking.

3 . The system of claim 2 , wherein the instructions, when executed, further cause the system to:

perform real-time image recognition on the image to detect the object and classify the object, based on the second light when the light source produces the second light; and

compare the object classification based on the amplified IR marking and the object classification based on the second light to produce a classification accuracy value;

in a case where the classification accuracy value is above a predetermined threshold:

generate a first bounding box around the detected object;

generate a first augmented view of the image based on the classification, the augmented view including the bounding box and a tag indicating the classification; and

display the augmented image on a display.

4 . The system of claim 3 , wherein in a case where the classification accuracy value is below the predetermined threshold, the instructions, when executed, further cause the system to display on the display an indication that the classification accuracy value is not within an expected range.

5 . The system of claim 4 , wherein the augmented view further includes an indication of the classification accuracy value.

6 . The system of claim 1 , wherein the IR marking includes a pattern that indicates a data bit and an error checking bit.

7 . The system of claim 6 , wherein the data bit indicates at least one of a surgical tool type or a surgical tool feature.

8 . The system of claim 1 , wherein the instructions, when executed, further cause the system to:

perform tracking of the detected object based on the IR marking.

9 . The system of claim 1 , wherein the instructions, when executed, further cause the system to:

adjust a control parameter of the system based on the detected object.

10 . The system of claim 1 , wherein the IR marking includes at least one of a logo, a two-dimensional code, a texture, a dot pattern, or a unique identifier.

11 . A method of object identification in endoscopy images, comprising:

providing light within a surgical operative site including at least one of a first light including an infrared (IR) band or a second light including a visible band;

switching the light source to produce only the first light;

capturing an image of an object within the surgical operative site, by an imaging device, the image including only the first light radiating from the object, the object including an infrared (IR) marking applied to the object, the IR marking including data corresponding to information about a type of the object;

accessing the image;

performing real-time image recognition on the image to detect the IR marking;

performing real-time image recognition on the image to detect the object and classify the object, based on the IR marking applied to the object;

generating an augmented image where the IR marking is removed from the image; and

displaying the augmented image on a display.

12 . The method of claim 11 , wherein the performing real-time image recognition on the image to detect the IR marking includes amplifying the IR marking.

13 . The method of claim 12 , the method further comprising:

performing real-time image recognition on the image to detect the object and classify the object, based on the second light when the light source produces the second light; and

comparing the object classification based on the amplified IR marking and the object classification based on the second light to produce a classification accuracy value;

in a case where the classification accuracy value is above a predetermined threshold:

generating a first bounding box around the detected object;

generating a first augmented view of the image based on the classification, the augmented view including the bounding box and a tag indicating the classification; and

displaying the augmented image on a display.

14 . The method of claim 13 , wherein in a case where the classification accuracy value is below the predetermined threshold, displaying on the display an indication that the classification accuracy value is not within an expected range.

15 . The method of claim 14 , wherein the augmented view further includes an indication of the classification accuracy value.

16 . The method of claim 15 , wherein the method further comprises performing tracking of the detected object based on the detected object based on the IR marking.

17 . The method of claim 11 , wherein the IR marking includes at least one of a logo, a two-dimensional code, a texture, a dot pattern, or a unique identifier.

18 . The method of claim 11 , wherein the IR marking includes a pattern that indicates a data bit and an error checking bit.

19 . The method of claim 18 , wherein the data bit indicates a surgical tool type.

20 . A non-transitory storage medium that stores a program causing a computer to execute a computer-implemented method of object identification in endoscopy images, the computer-implemented method comprising:

providing light within a surgical operative site including at least one of a first light including an infrared (IR) band or a second light including a visible band;

switching the light source to produce only the first light;

capturing an image of an object, by an imaging device, the image including only the first light radiating from the object, the object including an infrared (IR) marking applied to the object, the IR marking including data corresponding to information about a type of the object;

accessing the image;

performing real time image recognition on the image to detect the IR marking;

performing real time image recognition on the image to detect the object and classify the object, based on the IR marking applied to the object;

generating an augmented image where the IR marking is removed from the image; and

displaying the augmented image on a display.

Assignments (1)

Continuity (2)

References Cited (377)