Systems and methods for code parsing and lineage detection
Disclosed embodiments provide systems, methods, and techniques for lineage detection of calculations (e.g., processes, metrics, important data elements, data elements, or the like). According to disclosed embodiments, a lineage detector acquires one or more parameters, which at least include a first parameter indicative of source code and a second parameter indicative of a calculation. The lineage detector also parses the source code. After parsing the source code, the lineage detector determines the data lineage of the calculation based on the parsed source code. In some embodiments, the lineage detector generates and displays output of the data lineage.
1. A lineage detector, the lineage detector comprising:
a memory device storing instructions; and
a processor configured to execute the instructions to perform a method comprising:
acquiring a first parameter identifying a body of source code and a second parameter identifying a target calculation in the source code body;
parsing the body of the source code to determine elements of a query in the source code, the query including a “select all” statement, the parsing comprising:
identifying columns associated with a table referenced in the “select all” statement; and
modifying the “select all” statement to include the identified columns;
determining a data lineage of the target calculation using the elements of the query, the determination comprising:
generating a tree including hierarchical nodes containing the elements of the query;
identifying a first top node of the tree representing the target calculation parameter; and
identifying lower-level nodes having at least one of a direct or an indirect relationship to the first top node; and
generating an output of the determined data lineage using the first top node and the lower-level nodes.
2. The lineage detector of claim 1 , wherein the target calculation comprises at least one of a process, a metric, or a data element.
3. The lineage detector of claim 1 , wherein a top node of the nodes represents the target calculation.
4. The lineage detector of claim 1 , wherein a bottom node of the nodes represents a data element of the target calculation.
5. The lineage detector of claim 1 , wherein the processor is further configured to execute instructions to acquire a third parameter, a third parameter representing at least of:
a particular part of the target calculation;
a lineage focus time;
a characteristic associated with the target calculation; or
a detection level.
6. The lineage detector of claim 5 , wherein the processor is further configured to execute instructions to determine data lineage of the target calculation based on the third parameter.
7. The lineage detector of claim 1 , wherein the processor is further configured to execute instructions to retrieve the source code from a network location, wherein the first parameter identifies the network location.
8. The lineage detector of claim 1 , wherein the processor is further configured to execute instructions to display the generated output in an analyzer terminal.
9. The lineage detector of claim 1 , wherein the processor is further configured to display the generated output in a self-referential table.
10. The lineage detector of claim 1 , wherein the processor is further configured to execute instructions to:
acquire data from a database; and
determine the data lineage of the target calculation based on the data from the database.
11. The lineage detector of claim 10 , wherein a statement in the source code references an element in the database.
12. The lineage detector of claim 1 , wherein the displayed generated output is configured to be used by a software program.
13. The lineage detector of claim 1 , wherein the displayed generated output reflects a visual representation of data lineage.
14. The lineage detector of claim 1 , wherein the processor is further configured to execute instructions to use a software program to further parse the source code.
15. The lineage detector of claim 14 , wherein the processor is further configured to execute instructions to convert at least one of a string, a numeric data, and an SQL function, to conform the source code to requirements of the third-party software.
16. The lineage detector of claim 1 , wherein the processor is further configured to execute instructions to modify the source code, wherein modifying the source code comprises modifying a first column reflecting a column alias in the source code with the column name of a second column.
17. The lineage detector of claim 1 , wherein the processor is further configured to execute instructions to modify the source code, wherein modifying the source code comprises modifying a derived column in the source code to reflect a dependency on a base column.
18. The lineage detector of claim 1 , wherein the processor is further configured to execute instructions to modify the source code to link a first query in the source code to a second query in the source code.
19. The lineage detector of claim 1 , the parsing further comprising:
resolving an orphaned column in the source code by identifying a corresponding base column in the source code.
20. The lineage detector of claim 1 , the parsing further comprising:
detecting syntax errors in the source code that prevent the source code from executing.