Systems and methods for automated data dictionary generation and validation
Systems, apparatuses, methods, and computer program products are disclosed for automated generation and validation of a data dictionary. An example method includes automatically generating a data dictionary based on one or more reference term tokens of a database query. The example method further includes validating the data dictionary by generating, based on the database query and the data dictionary, a skeleton query, and determining, based on the skeleton query, a validation status of the data dictionary.
1. A method for automated data dictionary generation and validation, the method comprising:
receiving, by communications circuitry, a first data dictionary;
parsing, by data dictionary generation circuitry, a database query;
identifying, by the data dictionary generation circuitry and based on the parsing of the database query, one or more reference term tokens;
automatically generating, by the data dictionary generation circuitry, a second data dictionary based on the one or more reference term tokens, wherein automatically generating the second data dictionary comprises:
identifying, by the data dictionary generation circuitry, that a reference term token of the one or more reference term tokens in the database query does not include a data source name, and
storing, by the data dictionary generation circuitry, an indication of a missing data source name in a data source-referencing field of the second data dictionary;
validating, by data dictionary validation circuitry, the first data dictionary based on the second data dictionary by:
identifying, by the data dictionary validation circuitry, a set of reference term tokens included in both the first data dictionary and the second data dictionary, and
removing, by the data dictionary validation circuitry, the set of reference term tokens from the second data dictionary; and
causing, by the data dictionary validation circuitry, presentation of a validation status of the first data dictionary, wherein the validation status indicates an unsuccessful validation of the first data dictionary in an instance in which any of the one or more reference term tokens remain in the second data dictionary after removing the set of reference term tokens from the second data dictionary, and wherein the validation status indicates a successful validation of the first data dictionary in an instance in which no reference term tokens remain in the second data dictionary after removing the set of reference term tokens from the second data dictionary.
2. The method of claim 1 , wherein the first data dictionary is a manually-created data dictionary that was generated prior to automatic generation of the second data dictionary.
3. The method of claim 1 , wherein automatically generating the second data dictionary comprises:
converting, by the data dictionary generation circuitry, an alias indicated by a reference term token of the one or more reference term tokens in the database query to a data source name; and
storing, by the data dictionary generation circuitry, the data source name in a data source-referencing field of the second data dictionary.
4. The method of claim 1 , wherein automatically generating the second data dictionary comprises:
splitting, by the data dictionary generation circuitry, a respective reference term token based on a delimiter character of the respective reference term token; and
determining, by the data dictionary generation circuitry, a data source name and a field name for the respective reference term token.
5. The method of claim 1 , wherein the database query is a Structured Query Language (SQL) query.
6. An apparatus for automated data dictionary generation and validation, the apparatus comprising:
communications circuitry configured to receive a first data dictionary;
data dictionary generation circuitry configured to:
parse a database query,
identify, based on the parsing of the database query, one or more reference term tokens, and
automatically generate a second data dictionary based on the one or more reference term tokens, wherein automatically generating the second data dictionary comprises:
identifying that a reference term token of the one or more reference term tokens in the database query does not include a data source name, and
storing an indication of a missing data source name in a data source-referencing field of the second data dictionary;
data dictionary validation circuitry configured to:
validate the first data dictionary based on the second data dictionary by:
identifying a set of reference term tokens included in both the first data dictionary and the second data dictionary, and
removing the set of reference term tokens from the second data dictionary; and
cause presentation of a validation status of the first data dictionary, wherein the validation status indicates an unsuccessful validation of the first data dictionary in an instance in which any of the one or more reference term tokens remain in the second data dictionary after removing the set of reference term tokens from the second data dictionary, and wherein the validation status indicates a successful validation of the first data dictionary in an instance in which no reference term tokens remain in the second data dictionary after removing the set of reference term tokens from the second data dictionary.
7. The apparatus of claim 6 , wherein the first data dictionary is a manually-created data dictionary that was generated prior to automatic generation of the second data dictionary.
8. The apparatus of claim 6 , wherein the data dictionary generation circuitry is configured to automatically generate the second data dictionary by:
converting an alias indicated by a reference term token of the one or more reference term tokens in the database query to a data source name; and
storing the data source name in a data source-referencing field of the second data dictionary.
9. The apparatus of claim 6 , wherein the data dictionary generation circuitry is configured to automatically generate the second data dictionary by:
splitting a respective reference term token based on a delimiter character of the respective reference term token; and
determining a data source name and a field name for the respective reference term token.
10. The apparatus of claim 6 , wherein the database query is a Structured Query Language (SQL) query.
11. A computer program product for automated data dictionary generation and validation, the computer program product comprising at least one non-transitory computer-readable storage medium storing software instructions that, when executed, cause an apparatus to:
receive a first data dictionary;
parse a database query;
identify, based on the parsing of the database query, one or more reference term tokens;
automatically generate a second data dictionary based on the one or more reference term tokens, wherein automatically generating the second data dictionary comprises:
identifying that a reference term token of the one or more reference term tokens in the database query does not include a data source name; and
storing an indication of a missing data source name in a data source-referencing field of the second data dictionary;
validate the first data dictionary based on the second data dictionary by:
identifying a set of reference term tokens included in both the first data dictionary and the second data dictionary, and
removing the set of reference term tokens from the second data dictionary; and
cause presentation of a validation status of the first data dictionary, wherein the validation status indicates an unsuccessful validation of the first data dictionary in an instance in which any of the one or more reference term tokens remain in the second data dictionary after removing the set of reference term tokens from the second data dictionary, and wherein the validation status indicates a successful validation of the first data dictionary in an instance in which no reference term tokens remain in the second data dictionary after removing the set of reference term tokens from the second data dictionary.
12. The computer program product of claim 11 , wherein the first data dictionary is a manually-created data dictionary that was generated prior to automatic generation of the second data dictionary.
13. The computer program product of claim 11 , wherein the software instructions, when executed, further cause the apparatus to:
convert an alias indicated by a reference term token of the one or more reference term tokens in the database query to a data source name; and
store the data source name in a data source-referencing field of the second data dictionary.
14. The computer program product of claim 11 , wherein the software instructions, when executed, further cause the apparatus to:
split a respective reference term token based on a delimiter character of the respective reference term token; and
determine a data source name and a field name for the respective reference term token.
15. The method of claim 3 , wherein automatically generating the second data dictionary further comprises:
determining, by the data dictionary generation circuitry, the alias based on a native term token of the database query, wherein the native term token assigns the alias to the data source name; and
temporarily storing, by the data dictionary generation circuitry, a mapping of the alias to the data source name,
wherein the alias is converted to the data source name based on the mapping.
16. The method of claim 1 , wherein automatically generating the second data dictionary comprises:
identifying, by the data dictionary generation circuitry, a selection of all fields associated with a data source name in the database query; and
storing, by the data dictionary generation circuitry, an indication of the selection of all fields associated with the data source name in a field-referencing field of an entry in the second data dictionary.
17. The method of claim 16 , further comprising:
re-locating, by the data dictionary generation circuitry, the entry as a first entry in the second data dictionary.
18. The method of claim 1 , wherein validating the first data dictionary based on the second data dictionary is performed prior to transmission of data corresponding to data sources indicated by the first data dictionary from a first computing environment to a second computing environment.
19. The method of claim 1 , wherein causing presentation of the validation status of the first data dictionary comprises causing display of the validation status via a user interface of a client device.
20. The method of claim 1 , wherein parsing the database query comprises:
classifying, by the data dictionary generation circuitry, a respective token of the database query as a native term token or a reference term token.