Synthetic scene generation for autonomous vehicle testing
Aspects of the disclosed technology provide solutions for generating synthetic driving scenarios using text-based inputs that describe an intended operating goal. A process of the disclosed technology can include steps for generating a first synthetic scene for testing an autonomous vehicle (AV), providing the first synthetic scene to a first machine-learning model to generate a first text description of the synthetic scene, and providing the first text description to a second machine-learning model to determine if the first text description aligns with the predetermined operating goal for the AV. In some aspects, the process can further include generating a second text description for the synthetic scene, if the first text description does not align with the predetermined operating goal for the AV and providing the second text description to a third machine-learning model to generate a second synthetic scene. Systems and machine-readable media are also provided.
1. An apparatus comprising:
at least one memory; and
at least one processor coupled to the at least one memory, the at least one processor configured to:
generate a first synthetic scene for testing an autonomous vehicle (AV), wherein the first synthetic scene represents a three-dimensional (3D) environment that is based on a predetermined operating goal in relation to the AV;
provide the first synthetic scene to a first machine-learning model to generate a first text description of the first synthetic scene;
provide the first text description to a second machine-learning model to determine if the first text description aligns with the predetermined operating goal;
generate a second text description for the first synthetic scene, if the first text description does not align with the predetermined operating goal; and
provide the second text description to a third machine-learning model to generate a second synthetic scene, wherein the second synthetic scene corresponds to the predetermined operating goal for the AV.
2. The apparatus of claim 1 , wherein the at least one processor is further configured to:
determine a safety score for the second synthetic scene.
3. The apparatus of claim 2 , wherein the at least one processor is further configured to:
update an operating parameter of the AV based on the safety score.
4. The apparatus of claim 2 , wherein the safety score for the second synthetic scene is based on a safety score for the first synthetic scene.
5. The apparatus of claim 1 , wherein the predetermined operating goal specifies a set of regulatory requirements.
6. The apparatus of claim 1 , wherein the predetermined operating goal specifies a driving event.
7. The apparatus of claim 1 , wherein the predetermined operating goal specifies a safety violation.
8. The apparatus of claim 1 , wherein the first machine-learning model comprises a Generative Adversarial Network (GAN).
9. The apparatus of claim 1 , wherein the first synthetic scene is generated from a real driving scenario by matching characteristics of one or more objects in the first synthetic scene to one or more corresponding objects in the real driving scenario.
10. The apparatus of claim 1 , wherein the first and second synthetic scenes exhibit variations with respect to time.
11. The apparatus of claim 1 , wherein to generate the second text description the at least one processor is further configured to:
preserve one or more features of the first synthetic scene, while achieving the predetermined operating goal.
12. A computer-implemented method, comprising:
generating a first synthetic scene for testing an autonomous vehicle (AV), wherein the first synthetic scene represents a three-dimensional (3D) environment that is based on a predetermined operating goal in relation to the AV;
providing the first synthetic scene to a first machine-learning model to generate a first text description of the first synthetic scene;
providing the first text description to a second machine-learning model to determine if the first text description aligns with the predetermined operating goal;
generating a second text description for the first synthetic scene, if the first text description does not align with the predetermined operating goal for the AV; and
providing the second text description to a third machine-learning model to generate a second synthetic scene, wherein the second synthetic scene corresponds to the predetermined operating goal for the AV.
13. The computer-implemented method of claim 12 , further comprising:
determining a safety score for the second synthetic scene.
14. The computer-implemented method of claim 13 , further comprising:
updating an operating parameter of the AV based on the safety score.
15. The computer-implemented method of claim 13 , wherein the safety score for the second synthetic scene is based on a safety score for the first synthetic scene.
16. The computer-implemented method of claim 12 , wherein the predetermined operating goal specifies a set of regulatory requirements.
17. The computer-implemented method of claim 12 , wherein the predetermined operating goal specifies a driving event.
18. The computer-implemented method of claim 12 , wherein the predetermined operating goal specifies a safety violation.
19. The computer-implemented method of claim 12 , wherein the first machine-learning model comprises a Generative Adversarial Network (GAN).
20. A non-transitory computer-readable media comprising instructions stored thereon which, when executed are configured to cause a computer or processor to:
generate a first synthetic scene for testing an autonomous vehicle (AV), wherein the first synthetic scene represents a three-dimensional (3D) environment that is based on a predetermined operating goal in relation to the AV;
provide the first synthetic scene to a first machine-learning model to generate a first text description of the first synthetic scene;
provide the first text description to a second machine-learning model to determine if the first text description aligns with the predetermined operating goal;
generate a second text description for the first synthetic scene, if the first text description does not align with the predetermined operating goal for the AV; and
provide the second text description to a third machine-learning model to generate a second synthetic scene, wherein the second synthetic scene corresponds to the predetermined operating goal for the AV.