Granted Patent B2

US 12,636,098 · App. 18/028,130 · Granted May 26, 2026

Inventors: Abhilash K. Pandya (Grosse Ile, MI); Mostafa Daneshgar Rahbar (Dearborn Heights, MI); Luke A. Reisner (Madison Heights, MI); Hao Ying (Novi, MI); Mahendra Bhandari (Novi, MI); Madhusudhan Reddiboina (Southfield, MI)

Assignees: Wayne State University; RediMinds, Inc.

A61B34/30A61B5/02042G06T7/0016A61B2090/365G06T2207/10016G06T2207/10024G06T2207/20081

Abstract

Claims (85)

1 . A robotic surgical system, comprising:

a camera configured to capture a video of a surgical scene;

an output device configured to display the video;

at least one processor; and

memory storing instructions that, when executed by the at least one processor, cause the system to perform operations comprising:

identifying a first frame in the video;

identifying a second frame in the video;

identifying that the second frame depicts bleeding by:

determining a ratio of low-entropy red pixels in the first frame, the low-entropy red pixels in the first frame comprising pixels in the first frame with entropy levels over a first threshold and red channel values over a second threshold;

determining a ratio of low-entropy red pixels in the second frame, the low-entropy red pixels in the second frame comprising pixels in the second frame with entropy levels over the first threshold and red channel values over the second threshold; and

determining that a difference between the ratio of low-entropy pixels in the second frame and the ratio of low-entropy red pixels in the second frame is greater than a third threshold; and

based on identifying that the second frame depicts the bleeding, causing the output device to display the second frame with an augmentation indicating the bleeding.

2 . The robotic surgical system of claim 1 , comprising:

a console configured to control surgical tools, the surgical tools comprising:

a scope comprising the camera;

the output device; and

an input device configured to receive an input from a user, the console controlling the surgical tools based on the input.

3 . The robotic surgical system of claim 1 , wherein determining the ratio of low-entropy red pixels in the first frame comprises:

generating a first entropy mask representing local entropies of the pixels in the first frame by convolving an entropy kernel with a first detection window of the first frame;

generating a first masked frame by performing pixel-by-pixel multiplication of the first entropy mask and the first frame; and

identifying a number of pixels in the first masked frame with red channel values over the second threshold, and

wherein determining the ratio of low-entropy red pixels in the second frame comprises:

generating a second entropy mask representing local entropies of the pixels in the second frame by convolving the entropy kernel with a second detection window of the second frame;

generating a second masked frame by performing pixel-by-pixel multiplication of the second entropy mask and the second frame; and

identifying a number of pixels in the second masked frame with red channel values over the second threshold.

4 . A method, comprising:

identifying a first frame depicting a surgical scene;

identifying a second frame depicting the surgical scene;

identifying whether the second frame depicts bleeding by analyzing the first frame and the second frame; and

outputting the second frame with an augmentation indicating whether bleeding is depicted in the second frame;

wherein identifying whether the second frame depicts bleeding comprises:

generating a first entropy mask representing local entropies of first pixels in the first frame;

generating a second entropy mask representing local entropies of second pixels in the second frame; and

determining whether the second frame depicts bleeding based on the first entropy mask and the second entropy mask.

5 . The method of claim 4 , wherein generating the first entropy mask comprises applying an entropy kernel to the first frame; and

wherein generating the second entropy mask comprises applying the entropy kernel to the second frame.

6 . The method of claim 5 , wherein generating the first entropy mask comprises:

calculating a first local entropy of a first pixel in the first frame by convolving a first detection window with the entropy kernel, the first frame comprising the first detection window, the first detection window comprising the first pixel;

generating a first entropy pixel by comparing the first local entropy to a first threshold, a first value of the first entropy pixel being a first level or a second level based on whether the first local entropy is less than the first threshold; and

generating the first entropy mask to include the first entropy pixel, and wherein generating the second entropy mask comprises:

calculating a second local entropy of a second pixel in the second frame by convolving a second detection window with the entropy kernel, the second frame comprising the second detection window, the second detection window comprising the second pixel;

generating a second entropy pixel by comparing the second local entropy to the first threshold, a second value of the first entropy pixel being the first level or the second level based on whether the second local entropy is less than the first threshold; and

generating the second entropy mask to include the second entropy pixel.

7 . The method of claim 5 , wherein applying the entropy kernel to the first frame comprises convolving the entropy kernel with a first detection window of the first frame, and

wherein applying the entropy kernel to the second frame comprises convolving the entropy kernel with a second detection window of the second frame.

8 . The method of claim 4 , wherein determining whether the second frame depicts bleeding based on the first entropy mask and the second entropy mask comprises:

generating a first masked frame by performing pixel-by-pixel multiplication of the first entropy mask and the first frame;

identifying a first number of red pixels in the first masked frame;

generating a second masked frame by performing pixel-by-pixel multiplication of the second entropy mask and the second frame;

identifying a second number of red pixels in the second masked frame; and

determining whether the second frame depicts bleeding by comparing the first number and the second number.

9 . The method of claim 4 , wherein the first frame and the second frame are identified in multiple frames of a video, and

wherein the first frame and the second frame are nonconsecutive frames in the video.

10 . The method of claim 4 , wherein identifying whether the second frame depicts bleeding comprises determining that the second frame depicts the bleeding.

11 . The method of claim 10 , wherein outputting the second frame with the augmentation comprises:

identifying a portion of the first frame depicting a physiological structure obscured by the bleeding in the second frame, the first frame depicting the physiological structure without the bleeding; and

outputting the augmentation as a visual overlay of the second frame, the augmentation comprising the portion of the first frame.

12 . The method of claim 10 , further comprising:

determining a location of a source of the bleeding by:

identifying a region of the second frame depicting red pixels corresponding to less than a first threshold of local entropies, the region comprising a cluster of the red pixels; and

determining that the location of the source of the bleeding is within the region.

13 . The method of claim 10 , further comprising:

determining a location of a source of the bleeding, wherein determining the location of the source of the bleeding comprises determining a centroid of a region, the region being a largest cluster of red pixels corresponding to less than a first threshold of local entropies in the second frame.

14 . The method of claim 10 , further comprising:

determining a magnitude of the bleeding by:

identifying a region of the second frame depicting red pixels corresponding to lower than a first threshold of local entropies and red values greater than a threshold red value, the region comprising a cluster of the red pixels; and

determining the magnitude of the bleeding based on a change in an area of the region of the second frame and a corresponding area of a frame subsequent to the second frame.

15 . A robotic surgical system, comprising:

a tool comprising at least one sensor configured to generate a feedback signal indicating that the tool has touched a physiological structure;

at least one processor; and

memory storing instructions that, when executed by the at least one processor, cause the system to perform operations comprising:

identifying a first frame depicting a surgical scene;

identifying a second frame depicting the surgical scene;

identifying whether the second frame depicts bleeding by analyzing the first frame and the second frame and further based on the feedback signal; and

outputting the second frame with an augmentation indicating whether bleeding is depicted in the second frame.

16 . The robotic surgical system of claim 15 , further comprising:

a scope configured to obtain the first frame and the second frame; and/or

a display configured to output the first frame and the second frame.

17 . The robotic surgical system of claim 16 , further comprising: a tool comprising a 3-dimensional (3D) scanner,

wherein the operations further comprise: receiving, from the 3D scanner, volumetric data depicting the surgical scene.

18 . The robotic surgical system of claim 16 , further comprising: a tool comprising a camera configured to capture the first frame and the second frame,

wherein the operations further comprise: causing the tool to reposition based on whether the second frame depicts bleeding.

19 . The robotic surgical system of claim 16 , further comprising: one or more tools configured to stop the bleeding,

wherein the operations further comprise causing the one or more tools to stop the bleeding in the surgical scene.

20 . The robotic surgical system of claim 15 , further comprising a console configured to control one or more surgical tools, the one or more surgical tools comprising a scope configured to obtain the first frame and the second frame.

Assignments (2)

Continuity (2)

References Cited (49)