Mixed reality simulation system for testing vehicle control system designs
The disclosure includes a system, method, and tangible memory for generating a simulation. The method may include receiving real-world sensor stream data that describes a real-world data stream that is recorded by onboard sensors in one or more test vehicles, wherein the real-world sensor stream data describes real-world active agents and real-world environmental elements. The method may further include generating three-dimensional (3D) models of active agents and 3D models of environmental elements in a vehicle environment. The method may further include generating a simulation of the vehicle environment that synthesizes the real-world active agents with 3D models of active agents and synthesizes the real-world environmental elements with the 3D models of environmental elements.
1. A computing-implemented method comprising:
receiving real-world sensor stream data that describes real-world active agents and real-world environmental elements;
generating three-dimensional (3D) models of active agents and 3D models of environmental elements in a vehicle environment;
generating a simulation of the vehicle environment;
recording response data from a control system that is exposed to the simulation, wherein the control system is associated with an advanced driver assistance system (ADAS system); and
modifying a design of the ADAS system based on the response data by:
comparing the response data to a specification for the design of the ADAS system and identifying an operation performed by the ADAS system;
determining whether the response data is within a specified range as described by the specification; and
modifying the design for the operation based on the response data being outside the specified range.
2. The method of claim 1 , wherein:
the real-world sensor stream data describes a sensor stream for each onboard sensor in one or more test vehicles; and
the simulation is generated from mixed-reality streams that are a mix of the sensor stream for each of the onboard sensors in the one or more test vehicles with the 3D models of active agents and the 3D models of environmental elements.
3. The method of claim 2 , further comprising:
updating the mixed-reality streams for each sensor based on the response data.
4. The method of claim 1 , wherein comparing the response data to the specification includes determining how quickly the operation was performed.
5. The method of claim 1 , wherein:
the real-world sensor stream data further describes real-world active agent trajectories,
the method further comprises generating scripted behavior and mathematical models of active agent trajectories, and
generating the simulation includes synthesizing the real-world active agent trajectories with the scripted behavior and the mathematical models of active agent trajectories.
6. The method of claim 1 , further comprising:
providing initial conditions of the vehicle environment that includes the 3D models of active agents.
7. The method of claim 1 , further comprising:
comparing the response data to control system specifications to measure and assess an operation of the control system associated with an autonomous vehicle.
8. The method of claim 7 , further comprising:
modifying the ADAS system based on measurement and assessment of the operation of the control system associated with the ADAS system.
9. A system comprising:
one or more processors; and
a memory that stores instructions that, when executed by the one or more processors cause the one or more processors to perform operations comprising:
receiving real-world sensor stream data that describes real-world active agents and real-world environmental elements;
generating three-dimensional (3D) models of active agents and 3D models of environmental elements in a vehicle environment;
generating a simulation of the vehicle environment;
recording response data from a control system that is exposed to the simulation, wherein the control system is associated with an advanced driver assistance system (ADAS system); and
modifying a design of the ADAS system based on the response data by:
comparing the response data to a specification for the design of the ADAS system and identifying an operation performed by the ADAS system;
determining whether the response data is within a specified range as described by the specification; and
modifying the design for the operation based on the response data being outside the specified range.
10. The system of claim 9 , wherein:
the real-world sensor stream data describes a sensor stream for each onboard sensor in one or more test vehicles; and
the simulation is generated from mixed-reality streams that are a mix of the sensor stream for each of the onboard sensors in the one or more test vehicles with the 3D models of active agents and the 3D models of environmental elements.
11. The system of claim 10 , wherein the memory stores additional instructions that, when executed by the one or more processors cause the one or more processors to perform operations comprising:
updating the mixed-reality streams for each sensor based on the response data.
12. The system of claim 10 , wherein comparing the response data to a specification includes determining whether the operation protected a virtual pedestrian based on the specification for the design.
13. The system of claim 9 , wherein:
the real-world sensor stream data further describes real-world active agent trajectories,
the memory stores additional instructions that, when executed by the one or more processors cause the one or more processors to perform operations comprising generating scripted behavior and mathematical models of active agent trajectories, and
generating the simulation includes synthesizing the real-world active agent trajectories with the scripted behavior and the mathematical models of active agent trajectories.
14. The system of claim 9 , wherein the memory stores additional instructions that, when executed by the one or more processors cause the one or more processors to perform operations comprising:
providing initial conditions of the vehicle environment that includes the 3D models of active agents.
15. A non-transitory memory comprising computer code which, when executed by a processor, causes the processor to perform steps comprising:
receiving real-world sensor stream data that describes real-world active agents and real-world environmental elements;
generating three-dimensional (3D) models of active agents and 3D models of environmental elements in a vehicle environment;
generating a simulation of the vehicle environment that synthesizes;
recording response data from a control system that is exposed to the simulation, wherein the control system is associated with an advanced driver assistance system (ADAS system); and
modifying a design of the ADAS system based on the response data by:
comparing the response data to a specification for the design of the ADAS system and identifying an operation performed by the ADAS system;
determining whether the response data is within a specified range as described by the specification; and
modifying the design for the operation based on the response data being outside the specified range.
16. The non-transitory memory of claim 15 , wherein:
the real-world sensor stream data describes a sensor stream for each onboard sensor in one or more test vehicles; and
the simulation is generated from mixed-reality streams that are a mix of the sensor stream for each of the onboard sensors in the one or more test vehicles with the 3D models of active agents and the 3D models of environmental elements.
17. The non-transitory memory of claim 16 , wherein the steps further comprise:
updating the mixed-reality streams for each sensor based on the response data.
18. The non-transitory memory of claim 15 , wherein modifying the design for the operation includes modifying the operation to comply with the specification for the design.
19. The non-transitory memory of claim 15 , wherein:
the real-world sensor stream data further describes real-world active agent trajectories,
the steps further comprise generating scripted behavior and mathematical models of active agent trajectories, and
generating the simulation includes synthesizing the real-world active agent trajectories with the scripted behavior and the mathematical models of active agent trajectories.
20. The non-transitory memory of claim 15 , wherein the steps further comprise:
providing initial conditions of the vehicle environment that includes the 3D models of active agents.