Test vector generation from documentation
Aspects of the present disclosure relate to systems and methods that help automate the generation of a test vector. Every functional product, be it a service, device, or combination thereof, has one or more documents associated with it. These documents may include such documentation as: (1) Release Notes; (2) Configuration Guides; (3) command line interfaces (CLIs)/application program interfaces (APIs); (4) Data Sheets; (5) Installation Guides; (6) User Manuals; (7) Errata notices; and (8) other documentation. In embodiments, command data-models and ranges and sequence of valid inputs to the data-model are extracted from natural language expressions in documentation related to a specific product. In embodiments, this extracted information is used to generate one or more table of attributes and properties or constraints about which a test vector may be generated for testing a command.
1. A computer-implemented method for generating a test vector from technical documentation related to a product comprising:
generating a command template database (CT-DB) comprising one or more commands by extracting commands from the technical documentation using natural language processing;
generating, using the command template database, a command vector table (CVT) database comprising, for each command, at least some of its associated attributes and parameters;
generating, using, at least in part, location order of a command relative to another command or commands in the technical documentation and probabilistic relations based upon occurrences in the technical documentation, a command template sequence graph database that comprises representations of operational precedence ordered relationships of commands extracted from the technical documentation; and
Generating a test vector for a command selected from the command template database by:
Obtaining a command chain from the command template sequence database that identifies a set of commands for using the command and an ordering of commands in the set commands;
Obtaining parameters for the commands in the set of command from the command vector table database; and
Setting values for the parameters to form the test vector.
2. The computer-implemented method of claim 1 wherein a command comprises one or more of a command line interface (CLI) command, an application program interface (API), operation, call, query, script, or input.
3. The computer-implemented method of claim 1 wherein the step of generating a command template database (CT-DB) comprising one or more commands by extracting commands from the technical documentation using natural language processing comprises:
Extracting the one or more commands from the technical documentation related to the product;
Creating a command corpus of structured commands in which each command is in a structured format; and
Inputting the structured commands into an indexer for indexing.
4. The computer-implemented method of claim 1 wherein the step of generating, using the command template database, a command vector table database comprising, for each command, at least some of its associated attributes and parameters comprises:
For each command, parsing its corresponding command template from the CT-DB to tabulate at least some of its associated attributes and parameters into a CVT record.
5. The computer-implemented method of claim 1 wherein the step of generating, using, at least in part, location order of a command relative to another command or commands in the technical documentation and probabilistic relations based upon occurrences in the technical documentation, a command template sequence graph database that comprises representations of ordered relationships of commands extracted from the technical documentation comprises:
Creating a posting list for a set of commands from the technical documentation;
Generating frequency distributions of orderings in the posting list for the set of commands; and
Creating one or more sequence graphs of command ordering based upon probabilities of the frequency distributions of orderings.
6. The computer-implemented method of claim 5 wherein the set of commands are obtained from the command template database.
7. The computer-implemented method of claim 1 further comprising:
Using the test vector to verify information from technical documentation against actual functionality of the product.
8. The computer-implemented method of claim 7 wherein the step of using the test vector to verify information from technical documentation against actual functionality of the product comprising:
Using the test vector and one or more verification tools to verify information from technical documentation against actual functionality of the product.
9. The computer-implemented method of claim 8 wherein at least one of the one or more verification tools is an automated tool.
10. An information handling system comprising:
One or more processors;
A non-transitory computer-readable medium or media comprising one or more sets of instructions which, when executed by at least one of the one or more processors, causes steps to be performed comprising:
generating a command template database (CT-DB) comprising one or more commands by extracting commands from the technical documentation using natural language processing;
generating, using the command template database, a command vector table (CVT) database comprising, for each command, at least some of its associated attributes and parameters;
generating, using, at least in part, location order of a command relative to another command or commands in the technical documentation and probabilistic relations based upon occurrences in the technical documentation, a command template sequence graph database that comprises representations of operational precedence ordered relationships of commands extracted from the technical documentation; and
Generating a test vector for a command selected from the command template database by:
Obtaining a command chain from the command template sequence database that identifies a set of commands for using the command and an ordering of commands in the set commands;
Obtaining parameters for the commands in the set of command from the command vector table database; and
Setting values for the parameters to form the test vector.
11. The information handling system of claim 10 wherein a command comprises one or more of a command line interface (CLI) command, an application program interface (API), operation, call, query, script, or input.
12. The information handling system of claim 10 wherein the step of generating a command template database (CT-DB) comprising one or more commands by extracting commands from the technical documentation using natural language processing comprises:
Extracting the one or more commands from the technical documentation related to the product;
Creating a command corpus of structured commands in which each command is in a structured format; and
Inputting the structured commands into an indexer for indexing.
13. The information handling system of claim 10 wherein the step of generating, using the command template database, a command vector table database comprising, for each command, at least some of its associated attributes and parameters comprises:
For each command, parsing its corresponding command template from the CT-DB to tabulate at least some of its associated attributes and parameters into a CVT record.
14. The information handling system of claim 10 wherein the step of generating, using, at least in part, location order of a command relative to another command or commands in the technical documentation and probabilistic relations based upon occurrences in the technical documentation, a command template sequence graph database that comprises representations of ordered relationships of commands extracted from the technical documentation comprises:
Creating a posting list for a set of commands from the technical documentation;
Generating frequency distributions of orderings in the posting list for the set of commands; and
Creating one or more sequence graphs of command ordering based upon probabilities of the frequency distributions of orderings.
15. The information handling system of claim 14 wherein the set of commands are obtained from the command template database.
16. Information handling system of claim 10 wherein the non-transitory computer-readable medium or media further comprises one or more sets of instructions which, when executed by at least one of the one or more processors, causes steps to be performed comprising:
Using the test vector to verify information from technical documentation against actual functionality of the product.
17. A non-transitory computer-readable medium or media comprising one or more sequences of instructions which, when executed by at least one processor, causes steps to be performed comprising:
generating a command template database (CT-DB) comprising one or more commands by extracting commands from the technical documentation using natural language processing;
generating, using the command template database, a command vector table (CVT) database comprising, for each command, at least some of its associated attributes and parameters;
generating, using, at least in part, location order of a command relative to another command or commands in the technical documentation and probabilistic relations based upon occurrences in the technical documentation, a command template sequence graph database that comprises representations of functional precedence ordered relationships of commands extracted from the technical documentation; and
Generating a test vector for a command selected from the command template database by:
Obtaining a command chain from the command template sequence database that identifies a set of commands for using the command and an ordering of commands in the set commands;
Obtaining parameters for the commands in the set of command from the command vector table database; and
Setting values for the parameters to form the test vector.
18. The non-transitory computer-readable medium or media of claim 17 wherein the step of generating a command template database (CT-DB) comprising one or more commands by extracting commands from the technical documentation using natural language processing comprises:
Extracting the one or more commands from the technical documentation related to the product;
Creating a command corpus of structured commands in which each command is in a structured format; and
Inputting the structured commands into an indexer for indexing.
19. The non-transitory computer-readable medium or media of claim 17 wherein the step of generating, using the command template database, a command vector table database comprising, for each command, at least some of its associated attributes and parameters comprises:
For each command, parsing its corresponding command template from the CT-DB to tabulate at least some of its associated attributes and parameters into a CVT record.
20. The non-transitory computer-readable medium or media of claim 17 wherein generating, using, at least in part, location order of a command relative to another command or commands in the technical documentation and probabilistic relations based upon occurrences in the technical documentation, a command template sequence graph database that comprises representations of ordered relationships of commands extracted from the technical documentation comprises:
Creating a posting list for a set of commands from the technical documentation;
Generating frequency distributions of orderings in the posting list for the set of commands; and
Creating one or more sequence graphs of command ordering based upon probabilities of the frequency distributions of orderings.