Granted Patent B2

US 12,387,536 · App. 17/842,040 · Granted Aug 12, 2025

Inventors: Daniel Walker (Atherton, CA); Michael Jared Benisch (Menlo Park, CA)

Assignee: Woven by Toyota, Inc.

G07C5/0808G07C5/0841G07C5/0816

Abstract

Claims (67)

1. A system for assessing performance of an automated autonomous driving evaluator, the system comprising:

a processor; and

a memory storing machine-readable instructions that, when executed by the processor, cause the processor to:

receive a driving log that includes at least one of raw vehicle sensor data correlated with a time index and information derived from the raw vehicle sensor data correlated with the time index;

receive ground-truth data associated with the driving log, wherein the ground-truth data includes, for one or more respective values of the time index, at least one of human-initiated disengagements of an autonomous driving system and human-entered annotations indicating mistakes made by the autonomous driving system;

analyze the driving log using the automated autonomous driving evaluator to generate a report, wherein the automated autonomous driving evaluator is a computerized system that automatically analyzes driving logs to identify mistakes made by autonomous driving systems;

compare automatically the report with the ground-truth data to generate a performance assessment for the automated autonomous driving evaluator; and

provide the performance assessment to a development process in which the automated autonomous driving evaluator is modified based, at least in part, on the performance assessment to improve the performance of the automated autonomous driving evaluator;

wherein a vehicle is controlled using an autonomous driving system that has been corrected based on an analysis by the modified automated autonomous driving evaluator.

2. The system of claim 1 , wherein:

the ground-truth data is generated in an autonomous vehicle controlled by the autonomous driving system as the driving log is being recorded; and

a human observer riding in the autonomous vehicle initiates the human-initiated engagements and enters the human-entered annotations.

3. The system of claim 1 , wherein the ground-truth data is generated while the driving log is being replayed in a simulator by a human observer who enters the human-entered annotations.

4. The system of claim 1 , wherein the automated autonomous driving evaluator is designed to identify mistakes pertaining to a particular predetermined aspect of autonomous driving.

5. The system of claim 1 , wherein the performance assessment for the automated autonomous driving evaluator includes one or more of false positives, false negatives, true positives, and true negatives.

6. The system of claim 1 , wherein the driving log is recorded in an operating autonomous vehicle.

7. The system of claim 1 , wherein the driving log is generated synthetically by a computing system.

8. The system of claim 1 , wherein the automated autonomous driving evaluator is one of rule-based and machine-learning-based.

9. A system, comprising:

a processor; and

a memory storing machine-readable instructions that, when executed by the processor, cause the processor to:

receive a driving log that includes at least one of raw vehicle sensor data correlated with a time index and information derived from the raw vehicle sensor data correlated with the time index, wherein the driving log is recorded in an autonomous vehicle operating in a shadow mode in which an autonomous driving system simulates controlling the autonomous vehicle while the autonomous vehicle is being controlled by a human operator;

receive ground-truth data associated with the driving log, wherein simulated actions taken by the autonomous driving system in the shadow mode that deviate in a predetermined manner from actions taken by the human operator are automatically annotated as mistakes in the ground-truth data;

analyze the driving log using the automated autonomous driving evaluator to generate a report, wherein the automated autonomous driving evaluator is a computerized system that automatically analyzes driving logs to identify mistakes made by autonomous driving systems;

compare automatically the report with the ground-truth data to generate a performance assessment for the automated autonomous driving evaluator; and

provide the performance assessment to a development process in which the automated autonomous driving evaluator is modified based, at least in part, on the performance assessment to improve the performance of the automated autonomous driving evaluator;

wherein a vehicle is controlled using an autonomous driving system that has been corrected based on an analysis by the modified automated autonomous driving evaluator.

10. A non-transitory computer-readable medium for assessing performance of an automated autonomous driving evaluator and storing instructions that, when executed by a processor, cause the processor to:

receive a driving log that includes at least one of raw vehicle sensor data correlated with a time index and information derived from the raw vehicle sensor data correlated with the time index;

receive ground-truth data associated with the driving log, wherein the ground-truth data includes, for one or more respective values of the time index, at least one of human-initiated disengagements of an autonomous driving system and human-entered annotations indicating mistakes made by the autonomous driving system;

analyze the driving log using the automated autonomous driving evaluator to generate a report, wherein the automated autonomous driving evaluator is a computerized system that automatically analyzes driving logs to identify mistakes made by autonomous driving systems;

compare automatically the report with the ground-truth data to generate a performance assessment for the automated autonomous driving evaluator; and

provide the performance assessment to a development process in which the automated autonomous driving evaluator is modified based, at least in part, on the performance assessment to improve the performance of the automated autonomous driving evaluator;

wherein a vehicle is controlled using an autonomous driving system that has been corrected based on an analysis by the modified automated autonomous driving evaluator.

11. The non-transitory computer-readable medium of claim 10 , wherein:

the ground-truth data is generated in an autonomous vehicle controlled by the autonomous driving system as the driving log is being recorded; and

a human observer riding in the autonomous vehicle initiates the human-initiated engagements and enters the human-entered annotations.

12. A method, comprising:

receiving a driving log that includes at least one of raw vehicle sensor data correlated with a time index and information derived from the raw vehicle sensor data correlated with the time index;

receiving ground-truth data associated with the driving log, wherein the ground-truth data includes, for one or more respective values of the time index, at least one of human-initiated disengagements of an autonomous driving system and human-entered annotations indicating mistakes made by the autonomous driving system;

analyzing the driving log using an automated autonomous driving evaluator to generate a report, wherein the automated autonomous driving evaluator is a computerized system that automatically analyzes driving logs to identify mistakes made by autonomous driving systems;

comparing automatically the report with the ground-truth data to generate a performance assessment for the automated autonomous driving evaluator; and

providing the performance assessment to a development process in which the automated autonomous driving evaluator is modified based, at least in part, on the performance assessment to improve the performance of the automated autonomous driving evaluator;

wherein a vehicle is controlled using an autonomous driving system that has been corrected based on an analysis by the modified automated autonomous driving evaluator.

13. The method of claim 12 , wherein:

the ground-truth data is generated in an autonomous vehicle controlled by the autonomous driving system as the driving log is being recorded; and

a human observer riding in the autonomous vehicle initiates the human-initiated engagements and enters the human-entered annotations.

14. The method of claim 12 , wherein the ground-truth data is generated while the driving log is being replayed in a simulator by a human observer who enters the human-entered annotations.

15. The method of claim 12 , wherein the automated autonomous driving evaluator is designed to identify mistakes pertaining to a particular predetermined aspect of autonomous driving.

16. The method of claim 12 , wherein the performance assessment for the automated autonomous driving evaluator includes one or more of false positives, false negatives, true positives, and true negatives.

17. The method of claim 12 , wherein the driving log is recorded in an operating autonomous vehicle.

18. The method of claim 12 , wherein the driving log is generated synthetically by a computing system.

19. The method of claim 12 , wherein the automated autonomous driving evaluator is one of rule-based and machine-learning-based.

20. A method, comprising:

receiving a driving log that includes at least one of raw vehicle sensor data correlated with a time index and information derived from the raw vehicle sensor data correlated with the time index, wherein the driving log is recorded in an autonomous vehicle operating in a shadow mode in which an autonomous driving system simulates controlling the autonomous vehicle while the autonomous vehicle is being controlled by a human operator;

receiving ground-truth data associated with the driving log, wherein simulated actions taken by the autonomous driving system in the shadow mode that deviate in a predetermined manner from actions taken by the human operator are automatically annotated as mistakes in the ground-truth data;

analyzing the driving log using an automated autonomous driving evaluator to generate a report, wherein the automated autonomous driving evaluator is a computerized system that automatically analyzes driving logs to identify mistakes made by autonomous driving systems;

comparing automatically the report with the ground-truth data to generate a performance assessment for the automated autonomous driving evaluator; and

providing the performance assessment to a development process in which the automated autonomous driving evaluator is modified based, at least in part, on the performance assessment to improve the performance of the automated autonomous driving evaluator;

wherein a vehicle is controlled using an autonomous driving system that has been corrected based on an analysis by the modified automated autonomous driving evaluator.

21. A system, comprising:

a processor; and

a memory storing machine-readable instructions that, when executed by the processor, cause the processor to:

analyze, using an evaluator, a driving log to identify mistakes made by an autonomous driving system to generate a report of an analysis;

compare the report with ground-truth data to produce an assessment of the evaluator, wherein the ground-truth data includes at least one of human-initiated disengagements of the autonomous driving system and human-entered annotations indicating mistakes made by the autonomous driving system; and

communicate the assessment to a development process that modifies the evaluator to improve the performance of the evaluator;

wherein a vehicle is controlled using an autonomous driving system that has been corrected based on an analysis by the modified evaluator.

Assignments (2)

Continuity (1)

References Cited (13)