Generating synthetic driving scenes including temporary traffic control zones
Techniques are described herein for generating, using diffusion models and/or other generative models, synthetic driving scenes including traffic control zones. In various examples, a driving scene generator may provide the generative model with map data representing a driving environment, and input data representing attributes or objects of a traffic control zone (e.g., a construction zone). The diffusion model may be trained to populate the driving scene by generating agents and objects inside and outside of the traffic control zone in a realistic manner, consistent with the input data, to generate a synthetic driving scene. Synthetic driving scenes generated using generative models may be used to execute realistic simulations targeting specific driving scenarios that include construction zones and other traffic control zones.
1 . A system comprising:
one or more processors; and
one or more non-transitory computer-readable media storing computer-executable instructions that, when executed, cause the one or more processors to perform operations comprising:
receiving data representing a driving environment;
determining a first token representing an attribute of a first object associated with a temporary traffic control zone;
providing, to a diffusion model trained to output a synthetic driving scene, the data representing the driving environment and the first token;
generating, based at least in part on an output of the diffusion model, the synthetic driving scene including the temporary traffic control zone and the first object within the temporary traffic control zone, wherein the diffusion model is trained to generate the first object based at least in part on the first token; and
performing a driving simulation based at least in part on the synthetic driving scene.
2 . The system of claim 1 , the operations further comprising:
receiving a location of the temporary traffic control zone within the driving environment;
determining a second token representing the location of the temporary traffic control zone; and
providing the second token to the diffusion model, wherein the diffusion model determines a position of the first object within the temporary traffic control zone based at least in part on the second token.
3 . The system of claim 1 , wherein the first token represents a position and a size associated with the first object, and wherein the diffusion model determines a location and a geometric shape of the temporary traffic control zone based at least in part on the first token.
4 . The system of claim 1 , wherein:
the synthetic driving scene includes a top-down representation of the driving environment;
the top-down representation includes a channel storing a location of the temporary traffic control zone within the driving environment; and
performing the driving simulation comprises evaluating a behavior of a vehicle during the driving simulation, based at least in part on comparing a driving path of the vehicle to the location of the temporary traffic control zone.
5 . The system of claim 1 , wherein:
the first token represents an object type of the first object, the object type comprising at least one of a construction zone worker object type or a construction zone vehicle object type; and
performing the driving simulation comprises controlling the first object during the driving simulation, based at least in part on the object type.
6 . A method comprising:
receiving, by a scene generator, map data representing a driving environment;
receiving, by the scene generator, first input data associated with a temporary traffic control zone; and
generating, using a diffusion model trained to output a driving scene based at least in part on the map data, scene data representing the driving scene, wherein the driving scene includes the temporary traffic control zone and a first object associated with the temporary traffic control zone, and wherein the diffusion model is trained to generate the first object based at least in part on the first input data.
7 . The method of claim 6 , wherein the first input data represents a location of the temporary traffic control zone within the driving environment, and wherein the diffusion model determines a position of the first object based at least in part on the location of the temporary traffic control zone.
8 . The method of claim 6 , wherein the first input data represents a first object attribute associated with the first object, and wherein the diffusion model determines a location of the temporary traffic control zone based at least in part on the first object attribute.
9 . The method of claim 6 , wherein the first input data represents at least one of:
a driving scene attribute associated with the driving scene; or
a first object attribute associated with the first object, and
wherein the method further comprises:
determining a first token based at least in part on the first input data; and
providing the first token to the diffusion model, wherein the diffusion model is configured to generate the scene data using a diffusion process conditioned based at least in part on the first token.
10 . The method of claim 6 , wherein generating the first object comprises at least one of:
determining a position for the first object within the temporary traffic control zone; or
determining a first object attribute of the first object, wherein the first object attribute is associated with temporary traffic control zone.
11 . The method of claim 6 , wherein the driving scene includes a top-down representation of the driving environment, and wherein the top-down representation includes a channel storing a location of the temporary traffic control zone within the driving environment.
12 . The method of claim 11 , further comprising:
performing a driving simulation, based at least in part on the driving scene, to simulate potential interactions between a vehicle and the first object in the driving environment; and
evaluating a behavior of the vehicle during the driving simulation, based at least in part on comparing a driving path of the vehicle to the location of the temporary traffic control zone.
13 . The method of claim 6 , wherein the scene data includes a first object attribute representing an object type of the first object, wherein the object type comprises at least one of:
a traffic safety equipment object type;
a traffic control zone worker object type; or
a traffic control zone vehicle object type.
14 . The method of claim 13 , further comprising:
performing a driving simulation, based at least in part on the driving scene, to simulate potential interactions between a vehicle and the first object in the driving environment; and
controlling the first object during the driving simulation, based at least in part on the object type.
15 . One or more non-transitory computer-readable media storing instructions executable by a processor, wherein the instructions, when executed, cause the processor to perform operations comprising:
receiving, by a scene generator, map data representing a driving environment;
receiving, by the scene generator, first input data associated with a temporary traffic control zone; and
generating, using a diffusion model trained to output a driving scene based at least in part on the map data, scene data representing the driving scene, wherein the driving scene includes the temporary traffic control zone and a first object associated with the temporary traffic control zone, and wherein the diffusion model is trained to generate the first object based at least in part on the first input data.
16 . The one or more non-transitory computer-readable media of claim 15 , wherein the first input data represents a location of the temporary traffic control zone within the driving environment, and wherein the diffusion model determines a position of the first object based at least in part on the location of the temporary traffic control zone.
17 . The one or more non-transitory computer-readable media of claim 15 , wherein the first input data represents a first object attribute associated with the first object, and wherein the diffusion model determines a location of the temporary traffic control zone based at least in part on the first object attribute.
18 . The one or more non-transitory computer-readable media of claim 15 , wherein the first input data represents at least one of:
a driving scene attribute associated with the driving scene; or
a first object attribute associated with the first object, and
wherein the operations further comprise:
determining a first token based at least in part on the first input data; and
providing the first token to the diffusion model, wherein the diffusion model is configured to generate the scene data using a diffusion process conditioned based at least in part on the first token.
19 . The one or more non-transitory computer-readable media of claim 15 , wherein generating the first object comprises at least one of:
determining a position for the first object within the temporary traffic control zone; or
determining a first object attribute of the first object, wherein the first object attribute is associated with the temporary traffic control zone.
20 . The one or more non-transitory computer-readable media of claim 15 ,
wherein the driving scene includes a top-down representation of the driving environment, wherein the top-down representation includes a channel storing a location of the temporary traffic control zone within the driving environment, and wherein the operations further comprise:
performing a driving simulation, based at least in part on the driving scene, to simulate potential interactions between a vehicle and the first object in the driving environment; and
evaluating a behavior of the vehicle during the driving simulation, based at least in part on comparing a driving path of the vehicle to the location of the temporary traffic control zone.