System and methods thereof for monitoring proper behavior of an autonomous vehicle
A system and methods thereof for monitoring proper behavior of an autonomous vehicle are provided. The method includes generating a plurality of agents, wherein each of the plurality of agents describes a physical object, wherein at least one of the plurality of agents is an agent for the DUT, generating a plurality of scenarios, wherein each scenario models a behavior of at least one of the plurality of agents, and monitoring an interaction between the plurality of agents and the DUT agent for a scenario modeling the respective agent.
1. A method performed by a computer-implemented monitoring system for monitoring behavior of a simulated device under test (DUT), comprising:
receiving a data stream of an at least one physical object;
generating a plurality of agents, wherein each of the plurality of agents describes a physical object, wherein at least one of the plurality of agents is an agent for the DUT and wherein an agent for the at least one physical object is one of the plurality of agents, and wherein generating the agent for the at least one physical object is based on the received data stream;
generating a plurality of scenarios, wherein each scenario models a behavior of at least one of the plurality of agents;
monitoring an interaction between the plurality of agents other than the DUT agent and the DUT agent for a scenario modeling the agents;
generating an error indication based on the monitoring when there is a failure of the DUT agent to respond in a manner priorly defined as acceptable during at least one of the plurality of scenarios;
generating an alert corresponding to the error indication when the error indication is generated; and
supplying information to enable recreation of a sequence that led to the error indication when an error indication is generated.
2. The method of claim 1 , wherein the at least one physical object includes more than one physical object.
3. The method of claim 2 , further comprising:
generating at least a scenario for at least one object of the more than one physical objects.
4. The method of claim 1 , wherein the at least one physical object describes at least one of: a vehicle, a road, a sidewalk, a person, an animal, a traffic light, a traffic cone, a barrier, a bicycle, a train, and a weather element.
5. The method of claim 1 , wherein the DUT agent is an autonomous vehicle (AV).
6. The method of claim 1 , wherein a first scenario of the plurality of scenarios includes at least a second scenario of the plurality of scenarios.
7. The method of claim 5 , wherein a scenario of the plurality of scenarios describes at least one of: a cut-in in front of another vehicle, a cut from a left lane of another vehicle, a cut from a right lane of another vehicle, cuts from two lanes simultaneously, slowing in front of the autonomous vehicle, cutting and slowing in front of the DUT, and traffic light changes.
8. The method of claim 1 , further comprising:
generating, based on the monitoring, at least one of: a notification upon detection of a monitored event, a key performance indicator (KPI), and a coverage information.
9. The method of claim 1 , wherein the data stream comprises one or more video segments.
10. The method of claim 1 , further comprising:
executing by the computer-implemented monitoring system at least two of the plurality of agents in parallel.
11. The method of claim 1 , further comprising:
executing by the computer-implemented monitoring system at least two of the plurality of scenarios in parallel.
12. The method of claim 1 , wherein each scenario of the plurality of scenarios and each agent of the plurality of agents are described in a measurable scenario descriptive language (MSDL).
13. The method of claim 12 , wherein the MSDL is a declarative programming language for describing the behavior of the DUT and physical objects.
14. A non-transitory computer readable medium having stored thereon instructions for causing a processing circuitry to execute the method of claim 1 .
15. A system for monitoring behavior of a simulated device under test (DUT), comprising:
a network interface;
an input/output (I/O) interface;
a database;
a processing unit communicatively connected to the network interface, the I/O interface and the database, the processing unit being adapted to execute a plurality of instructions provided thereto;
a memory, a portion of which contains instructions for execution, wherein upon execution of the instructions by the processing unit, the monitoring system is adapted to:
receive a data stream of an at least one physical object;
generate a plurality of agents, wherein each of the plurality of agents describes a physical object, wherein at least one of the plurality of agents is an agent for the DUT and wherein an agent for the at least one physical object is one of the plurality of agents, and wherein generating the agent for the at least one physical object is based on the received data stream;
generate a plurality of scenarios, wherein each scenario models a behavior of at least one of the plurality of agents;
monitor an interaction between the plurality of agents other than the DUT agent and the DUT agent for a scenario modeling the agents;
generate an error indication based on the monitoring when there is a failure of the DUT agent to respond in a manner priorly defined as acceptable during at least one of the plurality of scenarios;
generate an alert corresponding to the error indication when the error indication is generate; and
supply information to enable recreation of a sequence that led to the error indication when an error indication is generated.
16. The system of claim 15 , wherein the at least one physical object includes more than one physical object.
17. The system of claim 16 , wherein the system is further configured to:
generate at least a scenario for at least one object of the more than one physical objects.
18. The system of claim 15 , wherein the at least one physical object describes at least one of: a vehicle, a road, a sidewalk, a person, an animal, a traffic light, a traffic cone, a barrier, a bicycle, a train and a weather element.
19. The system of claim 15 , wherein the DUT agent is an autonomous vehicle (AV).
20. The system of claim 15 , wherein a first scenario of the plurality of scenarios includes at least a second scenario of the plurality of scenarios.
21. The system of claim 19 , wherein a scenario of the plurality of scenarios describes at least one of: a cut-in in front of another vehicle, a cut from a left lane of another vehicle, a cut from a right lane of another vehicle, cuts from two lanes simultaneously, slowing in front of the autonomous vehicle, cutting and slowing in front of the DUT, and traffic light changes.
22. The system of claim 15 , wherein the system is further configured to:
generate, based on the monitoring, at least one of: a notification upon detection of a monitored event, a key performance indicator (KPI), and a coverage information.
23. The system of claim 15 , wherein the data stream comprises one or more video segments.
24. The system of claim 15 , wherein the system is further configured to:
execute by the system at least two of the plurality of agents in parallel.
25. The system of claim 15 , wherein the system is further configured to:
execute by the system at least two of the plurality of scenarios in parallel.
26. The system of claim 15 , wherein each scenario of the plurality of scenarios and each agent of the plurality of agents are described in a measurable scenario descriptive language (MSDL).
27. The system of claim 26 , wherein the DUT agent is an autonomous vehicle (AV) and wherein the MSDL is a declarative programming language for describing the behavior of the autonomous vehicle and physical objects.