IP Library › Granted Patent US 11,172,192
Granted Patent B2
US 11,172,192 · App. 16/377,527 · Granted Nov 9, 2021

Identifying defects in optical detector systems based on extent of stray light

Inventors: Chen David Lu (Campbell, CA); Matthew Rinehart (San Mateo, CA); Volker Grabe (Redwood City, CA)
Assignee: Waymo LLC
H04N17/002G06T7/0002G06T7/70H04N5/2351G06T2207/30168G06T2207/30252
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,172,192
App. No.
16/377,527
Granted
Nov 9, 2021
Kind
B2
Abstract

Example embodiments relate to identifying defects in optical detector systems based on extent of stray light. An example embodiment includes a method. The method includes capturing, using an optical detector system, an image of a scene that includes a bright object. The method also includes determining a location of the bright object within the image. Further, the method includes determining, based on the location of the bright object within the image, an extent of stray light from the bright object that is represented in the image. In addition, the method includes determining, by comparing the extent of stray light from the bright object that is represented in the image to a predetermined threshold extent of stray light, whether one or more defects are present within the optical detector system. The predetermined threshold extent of stray light corresponds to an expected extent of stray light.

Claims (63)

1. A method comprising:

capturing, using a camera, a first image of a scene that includes a bright object;

determining a location of the bright object within the first image;

determining, based on the location of the bright object within the first image, a first extent of diffuse veiling glare or lens flare from the bright object that is represented in the first image;

determining, by comparing the first extent of diffuse veiling glare or lens flare from the bright object that is represented in the first image to a predetermined threshold extent of stray light, whether one or more optical imperfections are present on an imaging optic of the camera,

wherein the predetermined threshold extent of stray light corresponds to an expected extent of stray light;

upon determining that one or more optical imperfections are present on the imaging optic of the camera, performing a corrective action;

after performing the corrective action, capturing, using the camera, a second image of the scene;

determining a second extent of diffuse veiling glare or lens flare represented in the second image; and

comparing the second extent of diffuse veiling glare to the first extent of diffuse veiling glare to determine whether the one or more optical imperfections are still present on the imaging optic of the camera.

2. The method of claim 1 , wherein the location of the bright object within the first image is determined based on a geographical location of the camera.

3. The method of claim 1 , wherein the one or more optical imperfections present on the imaging optic of the camera comprise condensation present on the imaging optic of the camera, and wherein the corrective action comprises wiping the imaging optic of the camera using a windshield wiper.

4. The method of claim 1 , wherein the corrective action comprises performing image analysis to alter images captured by the camera to mitigate errors caused by the one or more optical imperfections.

5. The method of claim 1 , wherein the one or more optical imperfections comprise an impurity in the imaging optic of the camera, a crack in the imaging optic of the camera, an air bubble in the imaging optic of the camera, a scratch on the imaging optic of the camera, dirt on the imaging optic of the camera, dust on the imaging optic of the camera, a bug splatter on the imaging optic of the camera, a degradation of a lens coating of the imaging optic of the camera, condensation on the imaging optic of the camera, debris on the imaging optic of the camera, imperfect transparency of the imaging optic of the camera, or warping of the imaging optic of the camera.

6. The method of claim 1 , wherein the first extent of diffuse veiling glare or lens flare from the bright object that is represented in the first image is also determined based on a predetermined angular size of the sun when observed from Earth.

7. The method of claim 1 , wherein taking corrective action comprises cleaning, repairing, or recalibrating.

8. The method of claim 1 , wherein determining whether one or more optical imperfections are present on the imaging optic of the camera comprises determining whether the imaging optic of the camera is unclean or occluded by debris.

9. The method of claim 8 , wherein determining whether the imaging optic within the camera is unclean or occluded by debris comprises determining whether:

the camera is facing backwards on a vehicle traveling forwards;

the camera is facing forwards on a vehicle traveling forwards;

the camera is facing backwards on a vehicle traveling backwards; or

the camera is facing forwards on a vehicle traveling backwards.

10. The method of claim 1 , wherein determining the first extent of diffuse veiling glare or lens flare from the bright object that is represented in the first image comprises:

identifying each pixel within the first image that has an intensity value above a predetermined threshold intensity value; and

generating a derivative image by converting all those pixels having an intensity value greater than the predetermined threshold intensity value to white and all those pixels having an intensity value less than or equal to the predetermined threshold intensity value to black.

11. The method of claim 10 , wherein determining the first extent of diffuse veiling glare or lens flare from the bright object that is represented in the first image further comprises determining a maximum radius of contiguous white pixels within the derivative image relative to a center of the location of the bright object.

12. The method of claim 10 , wherein determining the first extent of diffuse veiling glare or lens flare from the bright object that is represented in the first image further comprises identifying a secondary region of stray light that is separated by one or more black pixels from a primary region of stray light within the derivative image.

13. The method of claim 10 , wherein the predetermined threshold intensity value is based on a rejection ratio relative to an inherent luminance level of the bright object.

14. The method of claim 1 ,

wherein determining the first extent of diffuse veiling glare or lens flare from the bright object that is represented in the first image comprises identifying one or more streaks or stripes of stray light within the first image, and

wherein determining whether one or more optical imperfections are present on the imaging optic of the camera comprises identifying, based on the one or more streaks or stripes of stray light within the first image, that streaks of water or debris are present on the imaging optic of the camera as a result of motion of a windshield wiper across the imaging optic.

15. The method of claim 1 ,

wherein determining the first extent of diffuse veiling glare or lens flare from the bright object that is represented in the first image comprises identifying an existence of diffuse veiling glare within the first image, and

wherein determining whether one or more optical imperfections are present on the imaging optic of the camera comprises identifying, based on the diffuse veiling glare within the first image, that dust or water is present on the imaging optic of the camera.

16. The method of claim 1 ,

wherein determining the location of the bright object within the first image comprises capturing a baseline image of the scene at a reduced intensity,

wherein the baseline image is captured at a substantially similar perspective relative to the scene as the first image, and

wherein the baseline image is captured using a secondary camera that has a different dynamic range than the camera or using the camera with a modified dynamic range.

17. A non-transitory, computer-readable medium having instructions stored thereon, wherein the instructions, when executed by a processor, cause the processor to execute a method comprising:

receiving a first image of a scene that includes a bright object, wherein the first image was captured using a camera;

determining a location of the bright object within the first image;

determining, based on the location of the bright object within the first image, a first extent of diffuse veiling glare or lens flare from the bright object that is represented in the first image;

determining, by comparing the first extent of diffuse veiling glare or lens flare from the bright object that is represented in the first image to a predetermined threshold extent of stray light, whether one or more optical imperfections are present on an imaging optic of the camera,

wherein the predetermined threshold extent of stray light corresponds to an expected extent of stray light;

upon determining that one or more optical imperfections are present on the imaging optic of the camera, causing a corrective action to be performed;

after performing the corrective action, receiving a second image of the scene, wherein the second image was captured using the camera;

determining a second extent of diffuse veiling glare or lens flare represented in the second image; and

comparing the second extent of diffuse veiling glare to the first extent of diffuse veiling glare to determine whether the one or more optical imperfections are still present on the imaging optic of the camera.

18. A camera comprising:

an imaging optic;

an image sensor configured to receive light from a scene via the imaging optic; and

a controller configured to perform a monitoring routine, wherein the monitoring routine comprises:

capturing, using the camera, a first image of a scene that includes a bright object;

determining a location of the bright object within the first image;

determining, based on the location of the bright object within the first image, a first extent of diffuse veiling glare or lens flare from the bright object that is represented in the first image;

determining, by comparing the first extent of diffuse veiling glare or lens flare from the bright object that is represented in the first image to a predetermined threshold extent of stray light, whether one or more optical imperfections are present on an imaging optic of the camera,

wherein the predetermined threshold extent of stray light corresponds to an expected extent of stray light;

upon determining that one or more optical imperfections are present on the imaging optic of the camera, causing a corrective action to be performed;

after performing the corrective action, capturing, using the camera, a second image of the scene;

determining a second extent of diffuse veiling glare or lens flare represented in the second image; and

comparing the second extent of diffuse veiling glare to the first extent of diffuse veiling glare to determine whether the one or more optical imperfections are still present on the imaging optic of the camera.

19. The camera of claim 18 , wherein the controller is configured to perform the monitoring routine in response to a request from a computing device.

20. The camera of claim 19 , wherein the computing device is a server computing device configured to monitor states of multiple optical detector systems in a fleet of optical detector systems.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Apr 8, 2019
From: LU, CHEN DAVID; RINEHART, MATTHEW; GRABE, VOLKER
To: WAYMO LLC
Reel/Frame 048818/0418 →
Continuity (2)
Provisional Application 62785375 · Dec 27, 2018
Related Publication 20200213581A1 · Jul 2, 2020
Cited By (1)
US 12,489,988