Granted Patent B2

US 12,209,869 · App. 17/227,002 · Granted Jan 28, 2025

Inventors: Yi Xu (Pasadena, CA); Noureldin Ehab Hendy (West Lafayette, IN); Cooper Stokes Sloan (San Francisco, CA); Sarah Tariq (Palo Alto, CA); Feng Tian (Foster City, CA); Chuang Wang (Sunnyvale, CA)

Assignee: Zoox, Inc.

G01C21/30B60W60/001G01C21/3807G06N20/00B60W2420/403B60W2420/408B60W2552/53

Abstract

Claims (88)

1. A system comprising:

one or more processors; and

one or more non-transitory computer-readable media storing instructions executable by the one or more processors, wherein the instructions, when executed, cause the system to perform operations comprising:

receiving sensor data from a sensor associated with an autonomous vehicle in an environment, the sensor data comprising image data captured from a first perspective and lidar data captured from a second perspective;

inputting the sensor data to a machine learned model;

determining, by the machine learned model, lidar estimated map data representing the environment from a first top-down perspective and a first confidence value associated with the lidar estimated map data, wherein the first top-down perspective is different than the first perspective and the second perspective;

determining, by the machine learned model, image estimated map data representing the environment from a second top-down perspective and including a second confidence value;

receiving at least one of stored map data or trajectory corridor data determined based at least in part on the stored map data;

determining a first consistency score between the lidar estimated map data and the stored map data or trajectory corridor data;

determining a second consistency score between the image estimated map data and the stored map data or trajectory corridor data;

determining that the first consistency score meets or exceeds a first threshold and that the second consistency score meets or exceeds a second threshold; and

controlling an operation of the autonomous vehicle based at least in part on the first consistency score meeting or exceeding the first threshold, the second consistency score meeting or exceeding the second threshold, the first confidence value, and the second confidence value.

2. The system of claim 1 , wherein the sensor data is first image data associated with a first time, and wherein determining the image estimated map data comprises:

determining, based on the first image data, a third top-down representation of the environment;

determining, based at least in part on second image data of the environment associated with a second time, a fourth top-down representation of the environment;

determining, based at least in part on the fourth top-down representation and a difference in a pose of the autonomous vehicle between the first time and the second time, a fifth top-down representation of the environment; and

determining the image estimated map data based on the third top-down representation and the fifth top-down representation.

3. The system of claim 1 , wherein determining the second consistency score is based at least in part on:

an optical flow algorithm; and

determining a localization error.

4. The system of claim 1 , wherein the operations further comprise:

receiving at least one of the stored map data or the trajectory corridor data comprises receiving the trajectory corridor data; and

determining an overlap score between a location associated with the lidar estimated map data and a corresponding location associated with the trajectory corridor data.

5. The system of claim 4 , wherein the overlap score is further based at least in part on the first confidence value.

6. A method comprising:

receiving sensor data from a sensor associated with an autonomous vehicle in an environment, the sensor data being captured from a sensor perspective;

inputting the sensor data to a machine learned model;

receiving, from the machine learned model and based on inputting the sensor data into the machine learned model, first estimated map data representing a top-down representation of the environment and a confidence value associated with the first estimated map data, wherein the top-down representation is from a different perspective than the sensor perspective and the first estimated map data comprises one or more of roads or traffic control signals;

receiving at least one of stored map data or trajectory data determined based at least in part on the stored map data;

determining a first consistency score between the first estimated map data and the stored map data or trajectory corridor data;

determining a second consistency score between second estimated map data and the stored map data or trajectory corridor data;

determining that the first consistency score meets or exceeds a first threshold and that the second consistency score meets or exceeds a second threshold; and

determining an operation of the autonomous vehicle based at least in part on the first consistency score meeting or exceeding the first threshold and the second consistency score meeting or exceeding the second consistency score.

7. The method of claim 6 , wherein the sensor data is first image data associated with a first time, and wherein determining the first estimated map data comprises:

determining, based on the first image data, a first top-down representation of the environment;

determining, based at least in part on second image data of the environment associated with a second time, a second top-down representation of the environment;

determining, based at least in part on the second top-down representation and a difference in a pose of the autonomous vehicle between the first time and the second time, a third top-down representation of the environment; and

determining the first estimated map data based on the first top-down representation and the third top-down representation.

8. The method of claim 6 , wherein determining the first consistency score is based at least in part on:

an optical flow algorithm; and

the method further comprises determining a localization error.

9. The method of claim 6 , further comprising:

receiving at least one of the stored map data or the trajectory data comprises receiving the trajectory data; and

determining an overlap score between a location associated with the first estimated map data and a corresponding location associated with the trajectory data.

10. The method of claim 9 , wherein the overlap score is further based at least in part on the confidence value, which includes a per-pixel confidence value associated with the location.

11. The method of claim 9 , wherein the location associated with the first estimated map data comprises at least one of off-road data, on-road data, solid lane-line data, or dash lane-line data.

12. The method of claim 9 ,

wherein the overlap score is a first overlap score; and

wherein the determining the first consistency score comprises:

determining a second overlap score between a second location associated with the first estimated map data and a corresponding second location associated with the trajectory data, and

combining the first overlap score and the second overlap score.

13. The method of claim 6 ,

wherein the sensor data is first image data representing a first perspective of the environment at a first time;

wherein the method further comprises:

generating a first-perspective top-down representation associated with the first image data;

receiving second image data representing a second perspective of the environment at the first time; and

generating a second-perspective top-down representation associated with the second image data; and

wherein determining the first estimated map data comprises combining the first-perspective top-down representation with the second-perspective top-down representation.

14. The method of claim 6 ,

wherein the first estimated map data is based on a first top-down representation associated with a first time instant and a second top-down representation associated with a second time instant; and

wherein the machine learned model is trained at least in part on a cross-entropy loss based on the first top-down representation and the second top-down representation.

15. The method of claim 6 , wherein the sensor data comprises image data, the first estimated map data is first estimated map data, the method further comprising:

receiving lidar data from a lidar sensor associated with the autonomous vehicle;

determining second estimated map data based at least in part on the lidar data;

comparing, as a comparison, the second estimated map data to at least one of the stored map data or the trajectory data; and

determining the operation of the autonomous vehicle further based at least in part on the comparison.

16. The method of claim 6 , wherein determining the operation comprises at least one of remapping the environment or performing a different maneuver represented by an alternative proposed trajectory, which overlaps with portions of the first estimated map data having higher confidence values.

17. One or more non-transitory computer-readable media storing instructions that, when executed, cause one or more processors to perform operations comprising:

receiving first sensor data and second sensor data from at least one sensor associated with an autonomous vehicle in an environment, wherein the first sensor data is captured from a first perspective and the second sensor data is captured from a second perspective;

determining, based at least in part on the first sensor data and on a first pose, a first top-down representation of the environment;

determining, based at least in part on the second sensor data and a second pose, a second top-down representation of the environment;

determining, based at least in part on the first top-down representation and a difference between the first pose and the second pose, a third top-down representation of the environment, wherein the third top-down representation is from a different perspective than the first perspective or the second perspective;

determining, based on the second top-down representation and the third top-down representation, image estimated map data, wherein the image estimated map data comprises policy information associated with a rule of a road within the environment;

determining, by a machine learned model, lidar estimated map data;

receiving at least one of stored map data or trajectory corridor data determined based at least in part on the stored map data;

determining a first consistency score between the lidar estimated map data and the stored map data or trajectory corridor data;

determining a second consistency score between the image estimated map data and the stored map data or trajectory corridor data;

determining that the first consistency score meets or exceeds a first threshold and that the second consistency score meets or exceeds a second threshold; and

determining an operation of the autonomous vehicle based on the first consistency score meeting or exceeding the first threshold, the second consistency score meeting or exceeding the second threshold.

18. The one or more non-transitory computer-readable media of claim 17 ,

wherein the first sensor data comprises first image data and second image data;

wherein the operations further comprise:

generating a first-perspective top-down representation associated with the first image data;

generating a second-perspective top-down representation associated with the second image data; and

wherein generating the first top-down representation comprises combining the first-perspective top-down representation with the second-perspective top-down representation.

19. The one or more non-transitory computer-readable media of claim 17 , wherein the operations further comprise:

at least one of determining the operation of the autonomous vehicle, remapping the environment, or updating the stored map data.

20. The one or more non-transitory computer-readable media of claim 17 , wherein the lidar estimated map data is determined by inputting the first top-down representation and the third top-down representation into the machine learned model, which outputs the lidar estimated map data and confidence value data associated with the lidar estimated map data.

Assignments (1)

Continuity (1)

References Cited (34)

Cited By (1)

US 12,553,739