DOCUMENT VERIFICATION
Aspects of the present disclosure related to systems and methods that help automate the detection of errors in technical documentation. Every functional product, be it a service, device, or combination thereof, has one or more supporting documents associated with that product. These supporting documents may include such documentation as: (1) Release Notes; (2) Configuration Guides; (3) command line interfaces (CLIs)/application program interfaces (APIs); (4) Data Sheets; (5) Installation Guides; (6) User Manuals; (7) Errata notices; and (8) other documentation. It is important that the information provided in such documents, particularly the commands, be correct. In embodiments, a document verification system may be used to automatically extracted commands from technical documentation. And, in embodiments, these extracted commands and a definition set of commands may be compared to automatically detect errors in the documentation, which detected errors may checked and corrected before being released to customers.
1 . A computer-implemented method for detecting errors in technical documentation related to a product comprising:
generating a command template database (CT-DB) comprising one or more commands extracted from technical documentation related to the product;
generating a command context database (CC-DB) comprising representations of contextual relationships of data extracted from technical documentation related to the product; and
using the command template database (CT-DB) and command context database (CC-DB) to verify information from technical documentation against a structured data set associated with the product.
2 . The computer-implemented method of claim 1 wherein a command comprises one or more of a command line interface (CLI) command, an application program interface (API), operation, call, query, script, or input.
3 . The computer-implemented method of claim 1 wherein the step of generating a command template database (CT-DB) comprising one or more commands extracted from technical documentation related to the product comprises:
extracting the one or more commands from the technical documentation related to the product;
creating a command corpus of structured commands in which each command is in a structured format; and
inputting the structured commands into an indexer for indexing.
4 . The computer-implemented method of claim 1 wherein the step of generating a command context database (CC-DB) comprising representations of contextual relationships of data extracted from technical documentation related to the product comprises:
extracting text data from the technical documentation related to the product; and
vectorizing the text data to produce a trained model that comprises representations of contextual relationships between elements of at least some of the text data.
5 . The computer-implemented method of claim 1 wherein the step of using the command template database (CT-DB) and command context database (CC-DB) to verify information from technical documentation against a structured data set associated with the product comprises:
querying a query command obtained from the structured data set against the command template DB to find a set of matches;
responsive to the set of matches not comprising at least one exact match, logging that this query command is missing from documentation;
responsive to the set of matches comprising one or more close matches, logging the one or more close matches as potential errors for that command; and
outputting the logged results.
6 . The computer-implemented method of claim 5 further comprising the steps of:
for each close match that is logged:
performing a reverse lookup of the close match against a set of commands obtained from the structured data set; and
responsive to the close match having an exact match with a command in the set of commands obtained from the structured data set, removing the close match from the logged results.
7 . The computer-implemented method of claim 6 further comprising the steps of:
after any potential errors have been removed from the logged results, performing error classification on one or more of the errors in the logged results to identify a type of error.
8 . The computer-implemented method of claim 1 wherein the step of using the command template database (CT-DB) and command context database (CC-DB) to verify information from a document against a structured data set associated with the product comprises:
querying a query command obtained from the structured data set against the command context DB to determine whether a semantic mismatch exists; and
responsive to a semantic mismatch existing for the query command, logging the semantic mismatch for the query command.
9 . The computer-implemented method of claim 1 wherein the structured data set associated with the product is obtain by performing the steps comprising:
converting a definition data set associated with the product into a set of flattened plain text commands; and
for each of the flattened plain text commands, replacing delimiters or tokens with whitespace to obtain a natural language-like representation of the flattened plain text command.
10 . A documentation verification system for detecting errors in technical documentation related to a product, the system comprising:
a database generator that receives technical documentation related to the product and a definition data set related to the product and generates: ( 1 ) a command template database (CT-DB) comprising one or more commands extracted from technical documentation; ( 2 ) a command context database (CC-DB) comprising representations of contextual relationships of data extracted from the technical documentation; and ( 3 ) a set of normalized commands from the definition data set;
a command template classifier that uses at least part of the CT-DB to compare each query command from a set of query commands selected from the set of normalized commands to identify one or more errors in the set of query commands; and
a command context classifier that uses at least part of the CC-DB to perform a semantic relevance check on each query command from a set of query commands selected from the set of normalized commands to identify one or more errors in the set of query commands.
11 . The documentation verification system of claim 10 wherein the database generator generates the command template database (CT-DB) by performing the steps comprises:
extracting the one or more commands from the technical documentation related to the product;
creating a command corpus of structured commands in which each command is in a structured format; and
inputting the structured commands into an indexer for indexing.
12 . The documentation verification system of claim 10 wherein the database generator generates the command context database (CC-DB) by performing the steps comprises:
extracting text data from the technical documentation related to the product; and
vectorizing the text data to produce a trained model that comprises representations of contextual relationships between elements of at least some of the text data.
13 . The documentation verification system of claim 10 wherein the command template classifier that uses at least part of the CT-DB to compare each query command from a set of query commands selected from the set of normalized commands to identify one or more errors in the set of query commands by performing the steps comprises:
querying a query command obtained from the structured data set against the command template DB to find a set of matches;
responsive to the set of matches not comprising at least one exact match, logging that this query command is missing from documentation;
responsive to the set of matches comprising one or more close matches, logging the one or more close matches as potential errors for that command; and
outputting the logged results.
14 . The documentation verification system of claim 13 wherein the command template classifier further performs the steps comprising:
for each close match that is logged:
performing a reverse lookup of the close match against a set of commands obtained from the structured data set; and
responsive to the close match having an exact match with a command in the set of commands obtained from the structured data set, removing the close match from the logged results.
15 . The documentation verification system of claim 14 wherein the command template classifier further performs the steps comprising:
after any potential errors have been removed from the logged results, performing error classification on one or more of the errors in the logged results to identify a type of error.
16 . The computer-implemented method of claim 10 wherein the database generator generates the set of normalized commands from the definition data set by performing the steps comprising:
converting a definition data set associated with the product into a set of flattened plain text commands; and
for each of the flattened plain text commands, replacing delimiters or tokens with whitespace to obtain a natural language-like representation of the flattened plain text command.
17 . A non-transitory computer-readable medium or media comprising one or more sequences of instructions which, when executed by at least one processor, causes steps to be performed for detecting errors in technical documentation related to a product, the steps comprising:
generating a command template database (CT-DB) comprising one or more commands extracted from technical documentation related to the product;
generating a command context database (CC-DB) comprising representations of contextual relationships of data extracted from technical documentation related to the product; and
using the command template database (CT-DB) and command context database (CC-DB) to verify information from technical documentation against a structured data set associated with the product.
18 . The non-transitory computer-readable medium or media of claim 17 wherein the step of using the command template database (CT-DB) and command context database (CC-DB) to verify information from technical documentation against a structured data set associated with the product comprises:
querying a query command obtained from the structured data set against the command template DB to find a set of matches;
responsive to the set of matches not comprising at least one exact match, logging that this query command is missing from documentation;
responsive to the set of matches comprising one or more close matches, logging the one or more close matches as potential errors for that command; and
outputting the logged results.
19 . The non-transitory computer-readable medium or media of claim 18 wherein the non-transitory computer-readable medium or media further comprises one or more sequences of instructions which, when executed by at least one processor, causes steps to be performed comprising:
for each close match that is logged:
performing a reverse lookup of the close match against a set of commands obtained from the structured data set; and
responsive to the close match having an exact match with a command in the set of commands obtained from the structured data set, removing the close match from the logged results.
20 . The non-transitory computer-readable medium or media of claim 17 wherein the step of using the command template database (CT-DB) and command context database (CC-DB) to verify information from a document against a structured data set associated with the product comprises:
querying a query command obtained from the structured data set against the command context DB to determine whether a semantic mismatch exists; and
responsive to a semantic mismatch existing for the query command, logging the semantic mismatch for the query command.