Automated vehicle testing system based on requirements written in natural language
Provided are a method, system, and device for automated vehicle testing. The method may include, generating a ticket, wherein the ticket comprises at least one natural language testing requirement and at least one natural language incident scenario description; determining whether collected sensor data from a vehicle matches the at least one natural language incident scenario description in the ticket; based on determining that the collected sensor data from the vehicle matches the at least one natural language incident scenario description: generating a requirements as code (RaC) file based on the ticket and collected sensor data from a vehicle; and evaluating a ML model based on the RaC file to determine whether the ML model achieves the testing requirement, wherein the ML model is used to implement a vehicle application.
1 . A method for testing vehicle applications, the method comprising:
generating a ticket, wherein the ticket comprises at least one natural language incident scenario description;
determining whether collected sensor data from a vehicle matches the at least one natural language incident scenario description in the ticket; and
based on determining that the collected sensor data from the vehicle matches the at least one natural language incident scenario description:
generating a requirements as code (RaC) file based on the ticket and the collected sensor data from a vehicle; and
evaluating a ML model based on the RaC file to determine whether the ML model achieves a testing requirement, wherein the ML model is used to implement a vehicle application.
2 . The method as claimed in claim 1 , wherein the ticket is generated based on an incident record (IR) ticket which is written in a natural language.
3 . The method as claimed in claim 1 , wherein the ticket is stored in a ticket database, and the collected sensor data is obtained from a vehicle data database.
4 . The method as claimed in claim 1 , wherein determining whether the collected sensor data from the vehicle matches the at least one natural language incident scenario description in the ticket is performed using a sensor data classifier and/or a neural network.
5 . The method as claimed in claim 1 , wherein the RaC file comprises a ticket identifier, a file identifier, coded testing requirements, and a link to the collected sensor data.
6 . The method as claimed in claim 1 , wherein upon completion of evaluating the ML model based on the RaC file, the vehicle application is deployed into the vehicle.
7 . The method as claimed in claim 1 , wherein upon completion of evaluating the ML model based on the RaC file, a status of the ticket is updated based on a result of the evaluation.
8 . The method as claimed in claim 4 , wherein upon completion of generating the ticket, the ticket is received by a ticket receiver in the vehicle, and wherein the sensor data classifier and/or the neural network is implemented in the vehicle.
9 . The method as claimed in claim 8 , wherein based on determining that the collected sensor data from the vehicle matches the at least one natural language incident scenario description, the collected sensor data is transmitted by a data transmitter in the vehicle to a vehicle data database.
10 . An apparatus for testing vehicle applications, the apparatus comprising:
at least one memory storing computer-executable instructions; and
at least one processor configured to execute the computer-executable instructions to:
generate a ticket, wherein the ticket comprises at least one natural language incident scenario description;
determine whether collected sensor data from a vehicle matches the at least one natural language incident scenario description in the ticket; and
based on determining that the collected sensor data from the vehicle matches the at least one natural language incident scenario description:
generate a requirements as code (RaC) file based on the ticket and the collected sensor data from a vehicle; and
evaluate a ML model based on the RaC file to determine whether the ML model achieves a testing requirement, wherein the ML model is used to implement a vehicle application.
11 . The apparatus as claimed in claim 10 , wherein the ticket is generated based on an incident record (IR) ticket which is written in a natural language.
12 . The apparatus as claimed in claim 10 , wherein the ticket is stored in a ticket database, and the collected sensor data is obtained from a vehicle data database.
13 . The apparatus as claimed in claim 10 , wherein determining whether the collected sensor data from the vehicle matches the at least one natural language incident scenario description in the ticket is performed using a sensor data classifier and/or a neural network.
14 . The apparatus as claimed in claim 10 , wherein the RaC file comprises a ticket identifier, a file identifier, coded testing requirements, and a link to the collected sensor data.
15 . The apparatus as claimed in claim 10 , wherein upon completion of evaluating the ML model based on the RaC file, the vehicle application is deployed into the vehicle.
16 . The apparatus as claimed in claim 10 , wherein upon completion of evaluating the ML model based on the RaC file, a status of the ticket is updated based on a result of the evaluation.
17 . The apparatus as claimed in claim 13 , wherein upon completion of generating the ticket, the ticket is received by a ticket receiver in the vehicle, and wherein the sensor data classifier and/or the neural network is implemented in the vehicle.
18 . The apparatus as claimed in claim 17 , wherein based on determining that the collected sensor data from the vehicle matches the at least one natural language incident scenario description, the collected sensor data is transmitted by a data transmitter in the vehicle to a vehicle data database.