Automatic grammar based fault detection and isolation
View Patent ↗The present disclosure relates to automated testing of hardware and software systems. In some embodiments, a testing framework that generates a set of test cases for a system under test using a grammar is disclosed. The grammar may include terminal and nonterminal symbols with tags that describe a test sequence. The testing framework can use the grammar to generate a set of test cases. The testing framework can then receive feedback about the execution of the set of test cases from the system under test. In response to the feedback, the testing framework can generate a new set of grammars by automatically modifying or inserting tags in the original grammar. The new set of grammars can then be used to generate further test cases that intelligently explore the system under test for correctness, conformance, performance, security, or reliability.
1. A system for automating testing of control devices, the system comprising:
a computer memory that stores a grammar comprising one or more terminal and nonterminal symbols with tags that describe a first set of test cases; and
a processor that is operable to:
generates the first set of test cases for a device under test using the grammar;
receives feedback about the execution of the first set of test cases from the device under test;
in response to the feedback, automatically modify the grammar to identify at least one cause of failure of the device under test as a function of the received feedback; and
generate a second set of test cases for the device under test using the grammar.
2. The system of claim 1 , wherein the processor is further operable to modify tags of the grammar.
3. The system of claim 1 , wherein the processor is further operable to insert new tags into the grammar.
4. The system of claim 1 , wherein the processor is further operable to modify the grammar to isolate a cause of failure for the device under test.
5. The system of claim 1 , wherein the device under test comprises a programmable logic controller.
6. The system of claim 1 , wherein the processor modifies the grammar to reveal at least one cause of failure for the device under test.
7. The system of claim 1 , wherein the feedback identifies a fault in the device under test.
8. The system of claim 7 , wherein the processor is further operable to modify the grammar to identify a cause of or isolate the identified fault.
9. The system of claim 7 , wherein the processor is further operable to modify the grammar to remove at least one test case not associated with the identified fault.
10. The system of claim 1 , wherein the processor is further operable to generate a set of test cases using a grammar based on a generator configuration file.
11. The system of claim 10 , wherein the generator configuration file corresponds to at least one of a configuration option and a parameter for generating a set of test cases.
12. The system of claim 11 , wherein the processor automatically modifies the generator configuration file in response to the feedback.
13. A computer-implemented method of automatically generating tests using a grammar, the method comprising:
tagging a first grammar with a first set of tags;
creating a first set of test cases from the first grammar;
applying the first set of test cases to a system under test;
receiving output from the system under test;
based on the output, automatically creating a second grammar to identify at least one cause of failure of the system under test based; and
creating a second set of test cases from the second grammar.
14. The method of claim 13 , wherein the second grammar is created by modifying the first set of tags of the first grammar.
15. The method of claim 13 , wherein the second grammar is created by adding a second set of tags to the first grammar.
16. The method of claim 13 , further comprising applying the second set of test cases to the system under test.
17. The method of claim 13 , wherein the system under test comprises a hardware device.
18. The method of claim 13 , wherein the system under test comprises a software application.
19. The method of claim 13 , wherein the set of test cases simulates network attacks on the system under test.
20. The method of claim 13 , wherein automatically creating a second grammar includes automatically creating a second grammar to isolate a cause of failure for the device under test.
21. The method of claim 13 , wherein automatically creating a second grammar includes automatically creating a second grammar to reveal a cause of failure for the device under test.
22. The method of claim 13 , wherein the results identify a fault in the device under test.
23. The method of claim 22 , wherein automatically creating a second grammar includes automatically creating a second grammar to identify a cause of or isolate the identified fault.
24. A computer-implemented method of searching for fault revealing tests, the method comprising:
generating a first set of test cases from a grammar with tags for a system under test;
receiving results of the first set of test cases being applied to the system under test;
modifying the tags of the grammar in response to the results of the first set of test cases to isolate or reveal causes of failure of the system under test based on the received results; and
generating a second set of test cases from the grammar with tags for the system under test.
25. The method of claim 24 , wherein the system under test comprises a distributed control system.
26. The method of claim 24 , wherein the system under test comprises a supervisory control and data acquisition system.
27. The method of claim 24 , wherein modifying the tags of the grammar includes modifying the tags of the grammar to isolate a cause of failure for the device under test.
28. The method of claim 24 , wherein the modifying the tags of the grammar includes modifying the tags of the grammar to reveal a cause of failure for the device under test.
29. A computer readable medium having stored thereon computer executable components, the medium comprising:
a grammar framework that includes an original grammar with one or more tags that describes a set of one or more tests and generates the set of one or more tests from the original grammar; and
a grammar processing engine that receives results from the set of tests being executed on a system under test and, based on the results, automatically creates a new grammar with tags that describe tests that reveal faults of the system under test.
30. The computer readable medium of claim 29 , wherein the tests generated from the new grammar are smaller than the tests generated from the original grammar and reveal similar faults.
31. The computer readable medium of claim 29 , wherein the tests generated from the new grammar isolate a cause of failure for the system under test.
32. The computer readable medium of claim 29 , wherein the tests generated from the original grammar determine whether the system under test is connected to a network.
33. The computer readable medium of claim 29 , wherein the grammar processing engine automatically creates a new grammar with tags that describe tests that isolate faults of the system under test.