Granted Patent B2

US 12,299,853 · App. 17/966,455 · Granted May 13, 2025

Inventors: Ravish Malhotra (Oakland, CA); Kirk Gossage (Pacifica, CA)

Assignee: Verily Life Sciences LLC

G06T5/70A61B1/000095G06T5/50G06T7/0014G06T7/90G06T2207/10016G06T2207/10024G06T2207/10068G06T2207/20192G06T2207/20224G06T2207/30168

Abstract

Claims (90)

1. At least one non-transitory, machine-accessible storage medium that provides instructions that, when executed by a machine, will cause the machine to perform operations comprising:

receiving a video of a surgical scene, the video including an image frame including pixels each having a respective imaged color;

determining that the image frame includes a smoke occlusion, wherein the imaged color of the smoke occlusion combines an un-occluded color and a smoke color;

determining an estimated un-occluded color of one or more of the pixels using a mapping between a color space and a set of color bins including the estimated un- occluded color;

determining a respective estimated true color for each of the one or more pixels using the imaged color, the estimated un-occluded color, and the smoke color; and

generating a de-smoked image frame using the respective estimated true colors of the one or more pixels, the de-smoked image exhibiting a reduction of the smoke occlusion relative to the image frame,

wherein determining respective estimated true color for each of the one or more pixels comprises, for a pixel:

subtracting the smoke color from the imaged color to generate a normalized imaged color;

defining a tuple of spherical coordinates for the normalized imaged color in a spherical coordinate system having the smoke color as the origin; and

mapping the normalized imaged color to a respective estimated un-occluded color using the tuple of spherical coordinates.

2. The at least one non-transitory, machine-accessible storage medium of claim 1 , wherein generating the de-smoked image frame comprises, for the normalized pixel:

determining a first distance between the normalized imaged color and the smoke color along a smoke line between the smoke color and the estimated un-occluded color in the spherical coordinate system;

determining a transmission coefficient for the pixel as the first distance relative to a second distance between the smoke color and the estimated un-occluded color; and

generating the estimated true color using the transmission coefficient, the estimated un-occluded color, and the smoke color.

3. The at least one non-transitory, machine-accessible storage medium of claim 1 , wherein the image frame is a first image frame, and wherein the instructions, when executed by the machine, further cause the machine to execute operations comprising:

generating an average saturation value for a second image frame of the video;

generating a comparison of the average saturation value to an upper threshold value;

determining that the second image frame does not include surgical smoke using the comparison; and

generating the mapping using the second image frame.

4. The at least one non-transitory, one machine-accessible storage medium of claim 3 , wherein generating the mapping comprises:

defining a spherical coordinate system spanning the color space, wherein each point on a surface of a unit sphere in the spherical coordinate system corresponds to a distinct color in the color space;

populating a histogram by mapping each color in the second image frame to a bin of the plurality of uniformly distributed bins; and

determining a set of estimated un-occluded colors including the estimated un-occluded color using the histogram.

5. The at least one non-transitory, one machine-accessible storage medium of claim 3 , wherein the instructions, when executed by the machine, further cause the machine to execute operations comprising:

storing the mapping in a buffer of reference frame data, the buffer of reference frame data including mapping data from a plurality of reference frames preceding the second image frame in the video.

6. The at least one non-transitory, one machine-accessible storage medium of claim 1 , wherein the mapping is a first mapping and wherein the instructions, when executed by the machine, further cause the machine to execute operations comprising:

generating a second mapping of the color space to a second set of color bins including the smoke color using the image frame;

determining a plurality of smoke candidates using the second mapping;

generating a plurality of scores for the plurality of smoke candidates using the second mapping, each score of the plurality of scores representing a probability that a corresponding smoke candidate of the plurality of smoke candidates is the smoke color; and

replacing the smoke color with a smoke candidate from the plurality of smoke candidates using the plurality of scores.

7. The at least one non-transitory, machine-accessible storage medium of claim 1 , wherein determining that the image frame includes a smoke occlusion comprises:

generating an average luminance of the plurality of pixels of the image frame;

generating a comparison of the average luminance to a luminance threshold value; and

determining to process the image frame using the comparison.

8. The at least one non-transitory, machine-accessible storage medium of claim 7 , wherein determining to process the image frame using the comparison comprises:

generating an average saturation value for the plurality of pixels;

generating a comparison of the average saturation value to a saturation threshold value;

determining that the image frame includes the smoke occlusion using the comparison.

9. The at least one non-transitory, machine-accessible storage medium of claim 1 , wherein determining that the image frame includes a smoke occlusion comprises:

generating a set of principal color components using the plurality of pixels;

generating a comparison of the set of principal color components to a reference set of principal color components of a biological surface; and

determining to process the image frame using the comparison.

10. The at least one non-transitory, machine-accessible storage medium of claim 1 , wherein generating the de-smoked image frame comprises smoothing the de-smoked image using an edge-aware smoothing function.

11. The at least one non-transitory, one machine-accessible storage medium of claim 1 , wherein the instructions, when executed by the machine, further cause the machine to execute operations comprising:

generating visualization data for a display in communication with the machine using the de-smoked image frame; and

sending the visualization data to the display in place of the image frame in the video.

12. The at least one non-transitory, machine-accessible storage medium of claim 1 , wherein the video is a video stream, and wherein the machine executes the operations in near-real time.

13. A computer-implemented method, comprising:

receiving a video of a surgical scene, the video including an image frame including pixels each having a respective imaged color;

determining that the image frame includes a smoke occlusion, wherein the imaged color of the smoke occlusion combines an un-occluded color and a smoke color;

determining an estimated un-occluded color of one or more of the pixels using a mapping between a color space and a set of color bins including the estimated un-occluded color;

determining a respective estimated true color for each of the one or more pixels using the imaged color, the estimated un-occluded color, and the smoke color; and

generating a de-smoked image frame using the respective estimated true colors of the one or more pixels, the de-smoked image exhibiting a reduction of the smoke occlusion relative to the image frame,

wherein determining the respective estimated true color for each of the one or more pixels comprises, for a pixel:

subtracting the smoke color from the imaged color to generate a normalized imaged color;

defining a tuple of spherical coordinates for the normalized imaged color in a spherical coordinate system having the smoke color as the origin; and

mapping the normalized imaged color to a respective estimated un-occluded color using the tuple of spherical coordinates.

14. The computer-implemented method of claim 13 , wherein generating the de-smoked image frame comprises, for the normalized pixel:

determining a first distance between the normalized imaged color and the smoke color along a smoke line between the smoke color and the estimated un-occluded color in the spherical coordinate system;

determining a transmission coefficient for the pixel as the first distance relative to a second distance between the smoke color and the estimated un-occluded color; and

generating the estimated true color using the transmission coefficient, the estimated un-occluded color, and the smoke color.

15. The computer-implemented method of claim 13 , wherein the image frame is a first image frame, the method further comprising:

generating an average saturation value for a second image frame of the video;

generating a comparison of the average saturation value to an upper threshold value;

determining that the second image frame does not include surgical smoke using the comparison; and

generating the mapping using the second image frame.

16. The computer-implemented method of claim 13 , wherein the mapping is a first mapping, the method further comprising:

generating a second mapping of the color space to a second set of color bins including the smoke color using the image frame;

determining a plurality of smoke candidates using the second mapping;

generating a plurality of scores for the plurality of smoke candidates using the second mapping, each score of the plurality of scores representing a probability that a corresponding smoke candidate of the plurality of smoke candidates is the smoke color; and

replacing the smoke color with a smoke candidate from the plurality of smoke candidates using the plurality of scores.

17. The computer-implemented method of claim 13 , wherein determining that the image frame includes a smoke occlusion due comprises:

generating an average luminance of the plurality of pixels of the image frame;

generating a comparison of the average luminance to a luminance threshold value; and

determining to process the image frame using the comparison.

18. The computer-implemented method of claim 13 , wherein determining to process the image frame comprises:

generating a set of principal color components using the plurality of pixels;

generating a comparison of the set of principal color components to a reference set of principal color components of a biological surface; and

determining to process the image frame using the comparison.

19. At least one non-transitory, machine-accessible storage medium that provides instructions that, when executed by a machine, will cause the machine to perform operations comprising:

receiving a video of a surgical scene, the video including an image frame including pixels each having a respective imaged color;

determining that the image frame includes a smoke occlusion, wherein the imaged color of the smoke occlusion combines an un-occluded color and a smoke color;

determining an estimated un-occluded color of one or more of the pixels using a mapping between a color space and a set of color bins including the estimated un-occluded color;

determining a respective estimated true color for each of the one or more pixels using the imaged color, the estimated un-occluded color, and the smoke color; and

generating a de-smoked image frame using the respective estimated true colors of the one or more pixels, the de-smoked image exhibiting a reduction of the smoke occlusion relative to the image frame,

wherein the image frame is a first image frame, and wherein the instructions, when executed by the machine, further cause the machine to execute operations comprising:

generating an average saturation value for a second image frame of the video;

generating a comparison of the average saturation value to an upper threshold value;

determining that the second image frame does not include surgical smoke using the comparison; and

generating the mapping using the second image frame.

Assignments (2)

Continuity (3)

References Cited (29)