Granted Patent B2

US 12,307,636 · App. 17/731,127 · Granted May 20, 2025

Inventors: Nishant Verma (Burlingame, CA); Brian Anderson (Mountain View, CA)

Assignee: Verily Life Sciences LLC

G06T5/77G06T5/20G06T5/70G06T2207/10016G06T2207/10068

Abstract

Claims (139)

1. At least one non-transitory machine-accessible storage medium storing instructions that, when executed by a machine, will cause the machine to perform operations comprising:

receiving an image including a region of specular reflection;

detecting the region of specular reflection in the image;

estimating image information for a portion of the region of specular reflection, wherein the image information is extrapolated from peripheral pixels within a peripheral region surrounding the region of specular reflection and wherein the image information extrapolated from the peripheral pixels includes higher and lower frequency components; and

reconstructing the image with the image information populated into the region of specular reflection by reconstructing a central portion of the region of specular reflection based on less of, or a lower weighting of, the higher frequency components than used to reconstruct a peripheral portion of the region of specular reflection.

2. The at least one non-transitory machine-accessible storage medium of claim 1 , wherein reconstructing the image comprises replacing the specular reflection with the image information in the region.

3. The at least one non-transitory machine-accessible storage medium of claim 1 , wherein receiving the image comprises receiving a video stream comprising a plurality of video frames including the image.

4. The at least one non-transitory machine-accessible storage medium of claim 3 , wherein the receiving, the detecting, the estimating, and the reconstructing are performed in real-time for each video frame of the plurality of video frames while displaying the video stream.

5. The at least one non-transitory machine-accessible storage medium of claim 1 , wherein estimating image information comprises:

generating an image pyramid comprising a plurality of levels; and

extrapolating image intensity information for each level of the image pyramid.

6. The at least one non-transitory machine-accessible storage medium of claim 5 , wherein generating the image pyramid comprises:

applying a low-pass filter to a first level of the image pyramid to generate a filtered image; and

sub-sampling the filtered image to generate a downscaled image at a second level of the image pyramid.

7. The at least one non-transitory machine-accessible storage medium of claim 5 , wherein extrapolating image intensity information for each level of the image pyramid comprises:

generating a mapping of the region of specular reflection for each level of the image pyramid;

defining a missing region for each level of the image pyramid using the respective mapping; and

extrapolating image intensity information for each missing region.

8. The at least one non-transitory machine-accessible storage medium of claim 5 , wherein extrapolating image intensity information comprises, for a level “s” of the image pyramid:

generating an extrapolated intensity image, I s e (i, j), in accordance with the expression

I

S

e

⁡

(

i

,

j

)

=

I

S

⁡

(

i

,

j

)

*

G

σ

H

S

⁡

(

i

,

j

)

*

G

σ

where I s e represents the extrapolated intensity image at level “s” of the image pyramid, where G σ represents a gaussian smoothing kernel of width “σ,” and where H s (i, j) represents the mapping of the region of specular reflection at the level “s” of the image pyramid having “S” levels, wherein “i,” “j,” “s,” “σ,” and “S” are numerical values.

9. The at least one non-transitory machine-accessible storage medium of claim 1 , wherein reconstructing the image comprises:

defining a weight image for each of “S” extrapolated intensity images;

generating a reconstructed image by weighted combination of the extrapolated intensity images; and

generating an output image using the image and the reconstructed image.

10. The at least one non-transitory machine-accessible storage medium of claim 9 , wherein defining the weight image for an extrapolated intensity image comprises:

generating a cumulative weight image for each of the S extrapolated intensity images;

rescaling each cumulative weight image to a size of the image; and

combining the rescaled cumulative weight images.

11. The at least one non-transitory machine-accessible storage medium of claim 1 , wherein estimating the image information comprises:

generating an RGBA image from the image, wherein “R,” “G,” and “B,” represent red, green, and blue color channels, respectively, wherein A represents an alpha channel.

12. The at least one non-transitory machine-accessible storage medium of claim 11 , wherein reconstructing the image comprises, for an image pyramid of “S” levels:

defining an upper threshold, T UPPER and a lower threshold, T LOWER ; and

on a pixel-wise basis, for a value of the alpha channel being between T UPPER and T LOWER , determining a weight value for a level “s” of the pyramid and incrementing to a next lower level “s+1” of the image pyramid; or

for the value of the alpha channel being greater than T UPPER , combining image intensity information for each level of the image pyramid from a base level “0” to the level “s” of the image pyramid; or

for the value of the alpha channel being less than T LOWER , combining image intensity information for each level of the image pyramid from a base level to a level preceding the level “s” in the image pyramid.

13. The at least one non-transitory machine-accessible storage medium of claim 1 , wherein detecting the region of specular reflection comprises:

defining a pixel-wise luminance channel for the image;

comparing the luminance channel to a luminance threshold and

defining a binary mask for the image using the comparison.

14. The at least one non-transitory machine-accessible storage medium of claim 1 , wherein the image describes an inner space of a biological subject and wherein the image is captured by an endoscope.

15. The at least one non-transitory machine-accessible storage medium of claim 1 , wherein the instructions, when executed by the machine, cause the machine to perform further operations comprising:

outputting the reconstructed image to a display electronically coupled with the machine.

16. A computer-implemented method for reducing specular reflection in endoscopy images, the method comprising:

receiving an image including a region of specular reflection;

detecting the region of specular reflection in the image;

estimating image information for a portion of the region of specular reflection, wherein the image information is extrapolated from peripheral pixels within a peripheral region surrounding the region of specular reflection and wherein the image information extrapolated from the peripheral pixels includes higher and lower frequency components; and

reconstructing the image with the image information populated into the region of specular reflection by reconstructing a central portion of the region of specular reflection based on less of, or a lower weighting of, the higher frequency components than used to reconstruct a peripheral portion of the region of specular reflection.

17. The method of claim 16 , wherein reconstructing the image comprises replacing the specular reflection with the image information in the region.

18. The method of claim 16 , wherein receiving the image comprises receiving a video stream comprising a plurality of video frames including the image.

19. The method of claim 18 , wherein the receiving, the detecting, the estimating, and the reconstructing are performed in real-time for each video frame of the plurality of video frames while displaying the video stream.

20. The method of claim 16 , wherein estimating image information comprises:

generating an image pyramid comprising a plurality of levels; and

extrapolating image intensity information for each level of the image pyramid.

21. The method of claim 20 , wherein generating the image pyramid comprises:

applying a low-pass filter to a first level of the image pyramid to generate a filtered image; and

sub-sampling the filtered image to generate a downscaled image at a second level of the image pyramid.

22. The method of claim 20 , wherein estimating image information comprises:

generating a mapping of the region of specular reflection for each level of the image pyramid;

defining a missing region for each level of the image pyramid using the respective mapping; and

extrapolating image intensity information for each missing region.

23. The method of claim 20 , wherein extrapolating image intensity information comprises, for a level “s” of the image pyramid:

generating an extrapolated intensity image, I s e (i, j), in accordance with the expression

I

S

e

⁡

(

i

,

j

)

=

I

S

⁡

(

i

,

j

)

*

G

σ

H

S

⁡

(

i

,

j

)

*

G

σ

where I s e represents the extrapolated intensity image at level “s” of the image pyramid, where G σ represents a gaussian smoothing kernel of width “σ,” and where H s (i, j) represents the mapping of the region of specular reflection at the level “s” of the image pyramid having “S” levels, wherein “i,” “j,” “s,” “σ,” and “S” are numerical values.

24. The method of claim 20 , wherein reconstructing the image comprises, for “S” extrapolated intensity images:

defining a weight image for each level of the image pyramid;

generating a reconstructed image by weighted combination of the extrapolated intensity images; and

generating an output image using the image and the reconstructed image.

25. The method of claim 16 , wherein estimating the image information comprises:

generating an RGBA image from the image, wherein “R,” “G,” and “B,” represent red, green, and blue color channels, respectively, and wherein A represents an alpha channel.

26. The method of claim 25 , wherein reconstructing the image comprises, for an image pyramid of “S” levels:

defining an upper threshold, T UPPER and a lower threshold, T LOWER ; and

on a pixel-wise basis, for a value of the alpha channel being between T UPPER and T LOWER , determining a weight value for a level “s” of the pyramid and incrementing to a next lower level “s+1” of the image pyramid; or

for the value of the alpha channel being greater than T UPPER , combining image intensity information for each level of the image pyramid from a base level “0” to the level “s” of the image pyramid; or

for the value of the alpha channel being less than T LOWER , combining image intensity information for each level of the image pyramid from a base level to a level preceding the level “s” in the image pyramid.

Assignments (2)

Continuity (3)

References Cited (16)