Granted Patent B2

US 12,572,768 · App. 18/886,308 · Granted Mar 10, 2026

Inventors: Siddarth Satish (Portola Valley, CA); Kevin J. Miller (Mountain View, CA)

Assignee: Stryker Corporation

G06K7/146G06K7/1452G06K19/027G06K19/06037G06K19/0614G06T7/13

Abstract

Claims (54)

1 . A computer-implemented method for processing an image of a machine-readable code affixed to an item of a surgical system, the surgical system including a display and one or more processors, the method comprising:

accessing, with the one or more processors, the image that depicts the machine-readable code;

determining, with the one or more processors, the machine-readable code is not locatable within the image based on at least a portion of the machine-readable code being obscured, occluded, false, or blurry;

providing, with the one or more processors, the image as an input to a trained neural network;

refining, with the trained neural network, the image to render the machine-readable code locatable;

decoding, with the trained neural network, information coded within a machine-readable code region of the image, the decoded information including at least one of a size of the item, a manufacturer of the item, and a type of the item; and

displaying, on the display, fluid-related information based on the decoded information.

2 . The method of claim 1 , further comprising:

generating, with the trained neural network, a cleaned image of the machine-readable code; and

decoding, with the trained neural network, the information coded within the cleaned image.

3 . The method of claim 2 , further comprising locating, with the trained neural network, the machine-readable code region of the cleaned image of the machine-readable code.

4 . The method of claim 3 , further comprising generating, with the trained neural network, an adjusted image in which an orientation of the machine-readable code has been refined.

5 . The method of claim 4 , wherein the refined orientation of the machine-readable code includes determining a Hough transform of the cleaned image.

6 . The method of claim 4 , wherein the step of locating the machine-readable code region of the adjusted image includes performing at least one of corner detection or edge detection by:

applying a corner detection algorithm to the image to generate a heatmap including heatmap values corresponding to each coordinate within the image; and

detecting corners of the machine-readable code in the image based on the heatmap generated by the corner detection algorithm.

7 . The method of claim 5 , further comprising:

fitting a rectangle around corners of the machine-readable code in the adjusted image; and

estimating an orientation of the rectangle.

8 . The method of claim 2 , further comprising, with the trained neural network:

identifying a set of coordinates in the cleaned image as likely corresponding to the machine-readable code;

identifying at least one outlier coordinate of the set of coordinates that is unlikely to correspond to the machine-readable code; and

removing the at least one outlier coordinate to generate a cleaned set of coordinates.

9 . The method of claim 8 , wherein the step of decoding the information further comprises:

identifying the cleaned set of coordinates; and

analyzing a portion of the cleaned image which corresponds to the cleaned set of coordinates.

10 . The method of claim 1 , wherein adjusting the image of the machine-readable code includes at least one of:

reducing noise of the image, and increasing a signal to noise ratio of the image.

11 . The method of claim 1 , wherein the machine-readable code is realized as a color fiducial, and the decoded information further includes an assigned color value associated with the color fiducial.

12 . A computer-implemented method for processing an image of a machine-readable code affixed to an item of a surgical system, the surgical system including a display and one or more processors, the method comprising:

accessing, by the one or more processors, a preprocessed image that depicts the machine-readable code;

providing, with the one or more processors, the preprocessed image as an input to a trained neural network;

generating, with the trained neural network, an adjusted image in which an orientation of the machine-readable code has been refined;

decoding, with the trained neural network, information coded within a machine-readable code region of the adjusted image, the decoded information including at least one of a size of the item, a manufacturer of the item, and a type of the item; and

displaying, on the display, fluid-related information based on the decoded information.

13 . The method of claim 12 , wherein the step of generating the adjusted image includes performing at least one of corner detection or edge detection by:

applying a corner detection algorithm to generate a heatmap including heatmap values corresponding to each coordinate within the image; and

detecting corners of the machine-readable code based on the heatmap generated by the corner detection algorithm.

14 . The method of claim 13 , further comprising:

fitting a rectangle around the corners of the machine-readable code; and

estimating the orientation of the machine-readable code region by estimating an orientation of the rectangle.

15 . The method of claim 12 , wherein the machine-readable code is realized as a color fiducial, and the decoded information further includes an assigned color value associated with the color fiducial.

16 . A computer-implemented method for processing an image with a surgical system including a display and one or more processors, the method comprising:

receiving, at the one or more processors, a preprocessed image depicting a machine-readable code affixed to an item containing fluid including patient blood;

providing, with the one or more processors, the preprocessed image as an input to a trained neural network;

receiving, with the one or more processors and as an output from the trained neural network, a processed image of the machine-readable code;

decoding, with the one or more processors, information coded within the machine-readable code, the decoded information including at least one of a size of the item, a manufacturer of the item, and a type of the item; and

displaying, on the display, fluid-related information based on the decoded information.

17 . The method of claim 16 , wherein the decoded information further includes an item identifier.

18 . The method of claim 16 , wherein the machine-readable code comprises a color fiducial, and the decoded information further includes an assigned color value associated with the color fiducial.

19 . The method of claim 16 , further comprising estimating an orientation of the machine-readable code by determining a Hough transform of the processed image.

20 . The method of claim 16 , further comprising estimating an orientation of the machine-readable code:

applying a corner detection algorithm to the processed image to generate a heatmap including heatmap values corresponding to each coordinate within the image; and

detecting corners of the machine-readable code in the processed image based on the heatmap generated by the corner detection algorithm.

Assignments (2)

Continuity (4)

References Cited (65)