Granted Patent B1

US 12,275,427 · App. 17/137,103 · Granted Apr 15, 2025

Inventors: Arun Venkatraman (Mountain View, CA); James Andrew Bagnell (Pittsburgh, PA); Haoyang Fan (Redwood City, CA)

Assignee: GOOGLE LLC

B60W60/001B60W40/09B60W50/0097B60W50/06G05D1/0088B60W2510/18

Abstract

Claims (71)

1. A method for evaluating manual driving data using an autonomous vehicle control system by emulating an autonomous vehicle control system response using previously captured historical manual driving data, the method implemented by one or more processors and comprising:

capturing multiple instances of manual driving data, the multiple instances of manual driving data including at least a first instance of manual driving data and a second instance of manual driving data that are captured during control of a vehicle by a manual driver, wherein the second instance of manual driving data is captured succeeding the first instance of manual driving data, and wherein the second instance indicates a next trajectory of the vehicle that followed a trajectory of the vehicle from the first instance;

storing the multiple instances of manual driving data;

subsequent to the storing, for a first iteration of a plurality of iterations:

identifying the first instance of manual driving data from the stored multiple instances of manual driving data, the first instance of manual driving data being previously captured during control of the vehicle by the manual driver and comprising:

first vehicle trajectory data that defines one or more aspects of the trajectory of the vehicle controlled by the manual driver, and

corresponding first environmental data that defines one or more aspects of an environment of the vehicle for the first instance;

emulating the autonomous vehicle control system response using the previously captured manual driving data by processing the first instance of manual driving data as input, using the autonomous vehicle control system, to generate a predicted next instance of autonomous vehicle control system trajectory data defining one or more aspects of a predicted trajectory that is predicted to follow the trajectory from the first instance of manual driving data;

determining manual driving latency based upon the first instance of manual driving data, the second instance of manual driving data and the generated predicted next instance of autonomous vehicle control system trajectory data;

comparing (a) the predicted next instance of autonomous vehicle control system trajectory data to (b) the second instance of manual driving data, the second instance of manual driving data being previously captured during the control of the vehicle by the manual driver, and following the first instance of manual driving data, wherein the second instance of manual driving data comprises second vehicle trajectory data that defines one or more aspects of the next trajectory, of the vehicle controlled by the manual driver, that followed the trajectory from the first instance and wherein the comparing further is based upon the determined manual driving latency;

determining a difference measure based on the comparing; and

evaluating the manual driving data based on the difference measure from the plurality of iterations, comprising:

identifying, based on the difference measure from the plurality of iterations, one or more particular portions of the manual driving data to be excluded from the manual driving data,

and

providing the manual driving data that excludes the one or more particular portions, wherein the providing causes training of an additional autonomous vehicle control system.

2. The method of claim 1 , wherein the plurality of iterations capture manual driving data during control of the vehicle by the same manual driver, and wherein evaluating the manual driving data based on the difference measure from the plurality of iterations comprises evaluating the manual driving data based on the difference measure from the plurality of iterations.

3. The method of claim 1 wherein the training of the additional autonomous vehicle control system comprises training a machine learning model of the additional autonomous vehicle control system with the manual driving data that excludes the one or more particular portions.

4. The method of claim 1 wherein the step of evaluating the manual driving data includes determining at least one evaluation metric.

5. The method of claim 4 wherein the at least one evaluation metric results in exclusion of the one or more particular portions of the manual driving data for the training of the additional autonomous vehicle control system.

6. The method of claim 5 wherein a plurality of evaluation metrics are determined in evaluating the manual driving data.

7. The method of claim 6 wherein one of the plurality of evaluation metrics is a braking metric.

8. The method of claim 1 wherein evaluating the manual driving data includes determining a geographic evaluation metric.

9. The method of claim 8 wherein evaluating the manual driving data includes determining the geographic evaluation metric by evaluating the manual driving data against a plurality of autonomous vehicle control systems.

10. The method of claim 9 wherein each of the plurality of autonomous vehicle control systems are trained with one of a plurality of previously obtained geographically unique manual driving data sets.

11. The method of claim 1 , wherein the corresponding first environmental data that defines one or more aspects of the environment of the vehicle for the first instance is captured using a sensor suite of the vehicle.

12. The method of claim 1 , wherein the first vehicle trajectory data that defines the one or more aspects of the trajectory of the vehicle for the first instance includes one or more aspects of the trajectory of the vehicle for one or more previous instances.

13. The method of claim 1 , wherein the corresponding first environmental data that defines the one or more aspects of the environment of the vehicle for the first instance includes one or more aspects of the environment of the vehicle for one or more previous instances.

14. The method of claim 1 , further comprising:

determining whether to classify the second instance of manual driving data as a deviation, based on the difference measure.

15. The method of claim 14 , wherein the difference measure is further determined using a latency engine to account for delays in the manual driving data and based on latency of the manual driver.

16. The method of claim 14 , wherein determining whether to classify the second instance of manual driving data as a deviation based on the difference measure comprises:

determining a z-score value based on the second instance of manual driving data and the predicted next instance of autonomous vehicle control system trajectory, the predicted next instance of autonomous vehicle control system trajectory being a Gaussian distribution;

determining the z-score value satisfies one or more conditions; and

in response determining the z-score value satisfies the one or more conditions, determining to classify the second instance of manual driving data as a deviation.

17. The method of claim 15 , wherein determining whether to classify the second instance of manual driving data as a deviation based on the difference measure comprises:

determining a z-score value based on the second instance of manual driving data and the predicted next instance of autonomous vehicle control system trajectory, the predicted next instance of autonomous vehicle control system trajectory being a Gaussian distribution;

determining the z-score value does not satisfy one or more conditions; and

in response determining the z-score value does not satisfy the one or more conditions, determining to not classify the second instance of manual driving data as a deviation.

18. The method of claim 15 , wherein determining to classify the second instance of manual driving data as a deviation based on the difference measure comprises:

determining a log likelihood value based on the second instance of manual driving data and the predicted next instance of autonomous vehicle control system trajectory, the predicted next instance of autonomous vehicle control system trajectory being a Gaussian distribution;

determining the log likelihood value satisfies one or more conditions; and

in response determining the log likelihood value satisfies the one or more conditions, determining to classify the second instance of manual driving data as a deviation.

19. The method of claim 1 , wherein the evaluating includes determining if the difference measure excludes instances of manual driving data for being outside of a predetermined threshold delay.

20. A system including one or more processors that execute instructions, stored in an associated memory, the instructions when executed by the one or more processors evaluate manual driving data using an autonomous vehicle control system by emulating an autonomous vehicle control system response using previously captured historical manual driving data, the system comprising:

capturing multiple instances of manual driving data, the multiple instances of manual driving data including at least a first instance of manual driving data and a second instance of manual driving data that are captured during control of a vehicle by a manual driver, wherein the second instance of manual driving data is captured succeeding the first instance of manual driving data, and wherein the second instance indicates a next trajectory of the vehicle that followed a trajectory of the vehicle from the first instance;

storing the multiple instances of manual driving data;

subsequent to the storing, for a first iteration of a plurality of iterations:

identifying the first instance of manual driving data from the stored multiple instances of manual driving data, the first instance of manual driving data being previously captured during control of the vehicle by the manual driver and comprising:

first vehicle trajectory data that defines one or more aspects of the trajectory of the vehicle controlled by the manual driver, and

corresponding first environmental data that defines one or more aspects of an environment of the vehicle for the first instance;

emulating the autonomous vehicle control system response using the previously captured manual driving data by processing the first instance of manual driving data as input, using the autonomous vehicle control system, to generate a predicted next instance of autonomous vehicle control system trajectory data defining one or more aspects of a predicted trajectory that is predicted to follow the trajectory from the first instance of manual driving data;

determining manual driving latency based upon the first instance of manual driving data, the second instance of manual driving data and the generated predicted next instance of autonomous vehicle control system trajectory data;

comparing (a) the predicted next instance of autonomous vehicle control system trajectory data to (b) the second instance of manual driving data, the second instance of manual driving data being previously captured during the control of the vehicle by the manual driver, and following the first instance of manual driving data, wherein the second instance of manual driving data comprises second vehicle trajectory data that defines one or more aspects of the next trajectory, of the vehicle controlled by the manual driver, that followed the trajectory from the first instance and wherein the comparing further is based upon the determined manual driving latency;

determining a difference measure based on the comparing; and

evaluating the manual driving data based on the difference measure from the plurality of iterations, comprising:

identifying, based on the difference measure from the plurality of iterations, one or more particular portions of the manual driving data to be excluded from the manual driving data, and

providing the manual driving data that excludes the one or more particular portions, wherein the providing causes training of an additional autonomous vehicle control system.

21. A non-transitory computer-readable storage medium storing instructions executable by one or more processors of a computing system to evaluate manual driving data using an autonomous vehicle control system by:

capturing multiple instances of manual driving data, the multiple instances of manual driving data including at least a first instance of manual driving data and a second instance of manual driving data that are captured during control of a vehicle by a manual driver, wherein the second instance of manual driving data is captured succeeding the first instance of manual driving data, and wherein the second instance indicates a next trajectory of the vehicle that followed a trajectory of the vehicle from the first instance;

storing the multiple instances of manual driving data;

subsequent to the storing, for a first iteration of a plurality of iterations:

identifying the first instance of manual driving data from the stored multiple instances of manual driving data, the first instance of manual driving data being previously captured during control of the vehicle by the manual driver and comprising:

first vehicle trajectory data that defines one or more aspects of the trajectory of the vehicle controlled by the manual driver, and

corresponding first environmental data that defines one or more aspects of an environment of the vehicle for the first instance;

emulating an autonomous vehicle control system response using the previously captured manual driving data by processing the first instance of manual driving data as input, using the autonomous vehicle control system, to generate a predicted next instance of autonomous vehicle control system trajectory data defining one or more aspects of a predicted trajectory that is predicted to follow the trajectory from the first instance of manual driving data;

determining manual driving latency based upon the first instance of manual driving data, the second instance of manual driving data and the generated predicted next instance of autonomous vehicle control system trajectory data;

comparing (a) the predicted next instance of autonomous vehicle control system trajectory data to (b) the second instance of manual driving data, the second instance of manual driving data being previously captured during the control of the vehicle by the manual driver, and following the first instance of manual driving data, wherein the second instance of manual driving data comprises second vehicle trajectory data that defines one or more aspects of the next trajectory, of the vehicle controlled by the manual driver, that followed the trajectory from the first instance and wherein the comparing further is based upon the determined manual driving latency;

determining a difference measure based on the comparing; and

evaluating the manual driving data based on the difference measure from the plurality of iterations, comprising:

identifying, based on the difference measure from the plurality of iterations, one or more particular portions of the manual driving data to be excluded from the manual driving data, and

providing the manual driving data that excludes the one or more particular portions, wherein the providing causes training of an additional autonomous vehicle control system.

Assignments (3)

Continuity (1)

References Cited (44)