Granted Patent B2

US 12,561,942 · App. 18/540,461 · Granted Feb 24, 2026

Inventors: Xiaoying He (Palo Alto, CA); Haoxuan Zheng (Santa Clara, CA); Jingyuan Linda Zhang (Menlo Park, CA)

Assignee: Waymo LLC

G06V10/60G01S17/89G06V10/24

Abstract

Claims (119)

1 . A method comprising:

capturing, using a camera, an image of a first field of view of a surrounding environment from a first perspective;

capturing, using a light detection and ranging (lidar) device, a point cloud of a second field of view of the surrounding environment from a second perspective, wherein each point in the point cloud indicates a range between the lidar device and the surrounding environment in a region of the surrounding environment corresponding to the respective point;

aligning, by a computing device, the image with the point cloud based on the first field of view, the second field of view, the first perspective, or the second perspective;

identifying, by the computing device, one or more high-intensity regions of the surrounding environment based on the aligned image or point cloud;

identifying, by the computing device, one or more low-intensity regions of the surrounding environment based on the aligned image or point cloud;

determining, by the computing device, a first figure of merit that characterizes environmental conditions of the surrounding environment based on one or more intensity values of one or more portions of the image corresponding to at least one of the high-intensity regions of the surrounding environment; and

determining, by the computing device, a second figure of merit that characterizes environmental conditions of the surrounding environment by performing a fit based on:

at least one intensity value of at least one portion of the image corresponding to at least one of the low-intensity regions of the surrounding environment;

at least one distance value of at least one portion of the point cloud corresponding to at least one of the low-intensity regions of the surrounding environment; and

the first figure of merit.

2 . The method of claim 1 ,

wherein performing the fit comprising fitting data to an equation of the form of I(x)=I ∞ (1−e −range(x)/eMOR ),

wherein I(x) represents a light intensity at a particular region of the surrounding environment,

wherein the first figure of merit is I ∞ , and

wherein the second figure of merit is eMOR.

3 . The method of claim 1 ,

wherein the image of the first field of view of the surrounding environment from the first perspective is captured at a first wavelength,

wherein the first figure of merit and the second figure of merit characterize environmental conditions of the surrounding environment for light transmitted at the first wavelength, and

wherein the method further comprises:

capturing, using the camera, a second image of the first field of view of the surrounding environment from the first perspective, wherein the second image of the first field of view of the surrounding environment from the first perspective is captured at a second wavelength, and wherein the second wavelength is different from the first wavelength;

aligning, by the computing device, the second image with the point cloud based on the first field of view, the second field of view, the first perspective, or the second perspective;

identifying, by the computing device, one or more second high-intensity regions of the surrounding environment based on the aligned second image or point cloud;

identifying, by the computing device, one or more second low-intensity regions of the surrounding environment based on the aligned second image or point cloud;

determining, by the computing device, a third figure of merit that characterizes environmental conditions of the surrounding environment for light transmitted at the second wavelength based on one or more intensity values of one or more portions of the second image corresponding to at least one of the second high-intensity regions of the surrounding environment; and

determining, by the computing device, a fourth figure of merit that characterizes environmental conditions of the surrounding environment for light transmitted at the second wavelength by performing a second fit based on:

at least one intensity value of at least one portion of the second image corresponding to at least one of the second low-intensity regions of the surrounding environment;

at least one distance value of at least one portion of the point cloud corresponding to at least one of the second low-intensity regions of the surrounding environment; and

the third figure of merit.

4 . The method of claim 3 , further comprising:

capturing, using the camera, a third image of the first field of view of the surrounding environment from the first perspective, wherein the third image of the first field of view of the surrounding environment from the first perspective is captured at a third wavelength, wherein the third wavelength is different from the first wavelength, and wherein the third wavelength is different from the second wavelength;

aligning, by the computing device, the third image with the point cloud based on the first field of view, the second field of view, the first perspective, or the second perspective;

identifying, by the computing device, one or more third high-intensity regions of the surrounding environment based on the aligned third image or point cloud;

identifying, by the computing device, one or more third low-intensity regions of the surrounding environment based on the aligned third image or point cloud;

determining, by the computing device, a fifth figure of merit that characterizes environmental conditions of the surrounding environment for light transmitted at the third wavelength based on one or more intensity values of one or more portions of the third image corresponding to at least one of the third high-intensity regions of the surrounding environment; and

determining, by the computing device, a sixth figure of merit that characterizes environmental conditions of the surrounding environment for light transmitted at the third wavelength by performing a third fit based on:

at least one intensity value of at least one portion of the third image corresponding to at least one of the third low-intensity regions of the surrounding environment;

at least one distance value of at least one portion of the point cloud corresponding to at least one of the third low-intensity regions of the surrounding environment; and

the fifth figure of merit.

5 . The method of claim 4 ,

wherein the first wavelength is in the red portion of the visible electromagnetic spectrum,

wherein the second wavelength is in the blue portion of the visible electromagnetic spectrum, and

wherein the third wavelength is in the green portion of the visible electromagnetic spectrum.

6 . The method of claim 1 ,

wherein the image of the first field of view of the surrounding environment from the first perspective is captured at a first wavelength,

wherein the first figure of merit and the second figure of merit characterize environmental conditions of the surrounding environment for light transmitted at the first wavelength, and

wherein the method further comprises:

capturing, using the lidar device, a second point cloud of the second field of view of the surrounding environment from the second perspective, wherein the second point cloud of the second field of view of the surrounding environment from the second perspective is captured at a second wavelength, and wherein the second wavelength is different from the first wavelength;

identifying, by the computing device, one or more second high-intensity regions of the surrounding environment based on the second point cloud or the point cloud;

identifying, by the computing device, one or more second low-intensity regions of the surrounding environment based on the second point cloud or the point cloud;

determining, by the computing device, a third figure of merit that characterizes environmental conditions of the surrounding environment for light transmitted at the second wavelength based on one or more intensity values of one or more portions of the second point cloud corresponding to at least one of the second high-intensity regions of the surrounding environment; and

determining, by the computing device, a fourth figure of merit that characterizes environmental conditions of the surrounding environment for light transmitted at the second wavelength by performing a second fit based on:

at least one intensity value of at least one portion of the second point cloud corresponding to at least one of the second low-intensity regions of the surrounding environment;

at least one distance value of at least one portion of the point cloud corresponding to at least one of the second low-intensity regions of the surrounding environment; and

the third figure of merit.

7 . The method of claim 6 , wherein the second wavelength is in the near-infrared portion of the electromagnetic spectrum.

8 . The method of claim 6 , wherein capturing the second point cloud of the second field of view of the surrounding environment from the second perspective comprises recording, by the lidar device, near-infrared background light from the surrounding environment.

9 . The method of claim 1 ,

wherein the image of the first field of view of the surrounding environment from the first perspective is captured during a first time window,

wherein the point cloud of the second field of view of the surrounding environment from the second perspective is captured during the first time window,

wherein the first figure of merit and the second figure of merit characterize environmental conditions of the surrounding environment for light transmitted during the first time window, and

wherein the method further comprises:

capturing, using the camera, a second image of the first field of view of the surrounding environment from the first perspective, wherein the second image of the first field of view of the surrounding environment from the first perspective is captured during a second time window, and wherein the second time window is different from the first time window;

capturing, using the lidar device, a second point cloud of the second field of view of the surrounding environment from the second perspective, wherein each point in the second point cloud indicates a range between the lidar device and the surrounding environment in the region of the surrounding environment corresponding to the respective point, and wherein the second point cloud of the second field of view of the surrounding environment from the second perspective is captured during the second time window;

aligning, by the computing device, the second image with the second point cloud based on the first field of view, the second field of view, the first perspective, or the second perspective;

identifying, by the computing device, one or more second high-intensity regions of the surrounding environment based on the aligned second image or second point cloud;

identifying, by the computing device, one or more second low-intensity regions of the surrounding environment based on the aligned second image or second point cloud;

determining, by the computing device, a third figure of merit that characterizes environmental conditions of the surrounding environment for light transmitted during the second time window based on one or more intensity values of one or more portions of the second image corresponding to at least one of the second high-intensity regions of the surrounding environment; and

determining, by the computing device, a fourth figure of merit that characterizes environmental conditions of the surrounding environment for light transmitted during the second time window by performing a second fit based on:

at least one intensity value of at least one portion of the second image corresponding to at least one of the second low-intensity regions of the surrounding environment;

at least one distance value of at least one portion of the second point cloud corresponding to at least one of the second low-intensity regions of the surrounding environment; and

the third figure of merit.

10 . The method of claim 1 ,

wherein the second figure of merit characterizes the environmental conditions of the surrounding environment for a first angular direction with respect to the surrounding environment,

wherein the method further comprises determining, by the computing device, a third figure of merit that characterizes environmental conditions of the surrounding environment based on the performed fit, and

wherein the third figure of merit characterizes the environmental conditions of the surrounding environment for a second angular direction with respect to the surrounding environment.

11 . The method of claim 1 , wherein:

the first field of view is 360° in azimuth; or

the second field of view is 360° in azimuth.

12 . The method of claim 1 , wherein the one or more portions of the image corresponding to at least one of the low-intensity regions of the surrounding environment or at least one of the high-intensity regions of the surrounding environment are each one pixel in size.

13 . The method of claim 1 , wherein the one or more portions of the image corresponding to at least one of the low-intensity regions of the surrounding environment or at least one of the high-intensity regions of the surrounding environment are each multiple pixels in size.

14 . The method of claim 1 ,

wherein identifying the one or more high-intensity regions of the surrounding environment comprises identifying one or more portions of the image having a greatest intensity value, and

wherein identifying the one or more low-intensity regions of the surrounding environment comprises identifying one or more portions of the image having intensity values equal to less than a threshold percentage of the greatest intensity value.

15 . The method of claim 14 , wherein the threshold percentage is 1%.

16 . The method of claim 1 ,

wherein each point in the point cloud also indicates a reflectivity of an object in the surrounding environment that reflected a corresponding received lidar signal, and

wherein identifying the one or more low-intensity regions of the surrounding environment comprises identifying one or more portions of the point cloud having a reflectivity value that is less than a threshold reflectivity.

17 . The method of claim 1 , wherein identifying the one or more high-intensity regions of the surrounding environment comprises identifying one or more portions of the point cloud having a range value that is greater than a threshold range.

18 . The method of claim 1 , wherein aligning the image with the point cloud based on the first field of view, the second field of view, the first perspective or the second perspective comprises:

downsampling the image;

downsampling the point cloud;

adjusting an aspect ratio of the image;

adjusting an aspect ratio of the point cloud;

adjusting the image to account for parallax between the first perspective and the second perspective; or

adjusting the point cloud to account for parallax between the first perspective and the second perspective.

19 . A system comprising:

a camera configured to capture an image of a first field of view of a surrounding environment from a first perspective;

a light detection and ranging (lidar) device configured to capture a point cloud of a second field of view of the surrounding environment from a second perspective, wherein each point in the point cloud indicates a range between the lidar device and the surrounding environment in a region of the surrounding environment corresponding to the respective point;

a computing device configured to:

align the image with the point cloud based on the first field of view, the second field of view, the first perspective, or the second perspective;

identify one or more high-intensity regions of the surrounding environment based on the aligned image or point cloud;

identify one or more low-intensity regions of the surrounding environment based on the aligned image or point cloud;

determine a first figure of merit that characterizes environmental conditions of the surrounding environment based on one or more intensity values of one or more portions of the image corresponding to at least one of the high-intensity regions of the surrounding environment; and

determine a second figure of merit that characterizes environmental conditions of the surrounding environment by performing a fit based on:

at least one intensity value of at least one portion of the image corresponding to at least one of the low-intensity regions of the surrounding environment;

at least one distance value of at least one portion of the point cloud corresponding to at least one of the low-intensity regions of the surrounding environment; and

the first figure of merit.

20 . A non-transitory, computer-readable medium having instructions stored thereon, wherein the instructions, when executed by a processor, cause the processor to execute a method comprising:

receiving an image of a first field of view of a surrounding environment from a first perspective, wherein the image was captured using a camera;

receiving a point cloud of a second field of view of the surrounding environment from a second perspective, wherein the point cloud was captured using a light detection and ranging (lidar) device, and wherein each point in the point cloud indicates a range between the lidar device and the surrounding environment in a region of the surrounding environment corresponding to the respective point;

aligning the image with the point cloud based on the first field of view, the second field of view, the first perspective, or the second perspective;

identifying one or more high-intensity regions of the surrounding environment based on the aligned image or point cloud;

identifying one or more low-intensity regions of the surrounding environment based on the aligned image or point cloud;

determining a first figure of merit that characterizes environmental conditions of the surrounding environment based on one or more intensity values of one or more portions of the image corresponding to at least one of the high-intensity regions of the surrounding environment; and

determining a second figure of merit that characterizes environmental conditions of the surrounding environment by performing a fit based on:

at least one intensity value of at least one portion of the image corresponding to at least one of the low-intensity regions of the surrounding environment;

at least one distance value of at least one portion of the point cloud corresponding to at least one of the low-intensity regions of the surrounding environment; and

the first figure of merit.

Assignments (1)

Continuity (1)

References Cited (34)