Design-time information based on run-time artifacts in a distributed computing cluster
Techniques are disclosed for inferring design-time information based on run-time artifacts generated by services operating in a distributed computing cluster. In an embodiment, a metadata system extracts metadata including run-time artifacts generated by services in a distributed computing cluster while processing a workflow including multiple jobs. The extracted metadata is processed to identify entities and entity relationships which can then be used to generate lineage information. Using the lineage information, the metadata system can infer design-time information associated with the workflow. The inferred design-time information can then be utilized to, for example, recreate the workflow, recreate previous versions of the workflow, optimize the workflow, etc.
1. A method comprising:
receiving metadata from a computer system, the metadata generated by the computer system on execution, of a workflow including one or more data processing jobs, the metadata including run-time artifacts generated at a time of executing the workflow;
determining a data lineage associated with the workflow based on the metadata;
generating design-time information associated with the workflow based on the data lineage, the design-time information indicative of a design of any of the computer system, the workflow, or the one or more data processing jobs included in the workflow; and
generating a data lineage visualization based on the data lineage associated with the workflow; and
causing display of the data lineage visualization.
2. The method of claim 1 , wherein determining the data lineage associated with the workflow includes:
identifying, based on the metadata, a plurality of entities involved in executing the one or more data processing jobs includes in the workflow; and
determining, based on the metadata, relationships between the plurality of entities.
3. The method of claim 2 , wherein the plurality of entities include any one or more of a file, a directory, a table, a script, a query template, a query execution object, a job template, or a job execution object.
4. The method of claim 2 , wherein the relationships between entities include any one or more of a data flow relationship, a parent-child relationship, a logical-physical relationship, or a control relationship.
5. The method of claim 1 , wherein the run time metadata is extracted from one or more services for storing and processing data in the computer system.
6. The method of claim 1 , further comprising:
determining a structure of a previous version of the workflow based on the design-time information.
7. The method of claim 1 , wherein the design-time information includes information regarding any one or more of:
data processed by the workflow;
operations performed on the data as part of the workflow; or
services of the computer system utilized to perform the operations on the data.
8. The method of claim 1 , wherein generating the design-time information includes:
inferring, based on the data lineage associated with the workflow, logical connections between the one or more data processing jobs included in the workflow.
9. The method of claim 8 , wherein logical connections between data processing jobs may include any one or more of:
sequencing of data processing jobs;
scheduling of data processing jobs;
dependencies between data processing jobs; or
common parameters between data processing jobs.
10. The method of claim 1 , the method further comprising optimizing a structure of the workflow, wherein optimizing the structure of the workflow includes any one or more of:
changing the data processed according to the workflow;
changing the sequencing and/or scheduling of data processing jobs included in the workflow; or
changing one or more of the services utilized to store and process the data according to the workflow.
11. The method of claim 1 , the method further comprising optimizing a structure of the workflow, wherein the structure of the workflow is optimized to improve data processing efficiency and/or data storage efficiency.
12. The method of claim 1 , further comprising:
configuring the computer system to log and prepare the metadata for extraction.
13. The method of claim 1 , wherein the workflow is heterogeneous, the heterogeneous workflow including a plurality of different types of data processing jobs performed by a plurality of different types of services associated with the computer system.
14. The method of claim 1 , further comprising:
determining if the workflow involves processing personally identifiable information (PII) based on the design-time information.
15. The method of claim 1 , wherein the data lineage visualization includes: a plurality of graphical entity nodes representative of a least some of a plurality of the entities involved in the processing of data according to the workflow, each of the plurality of graphical entity nodes visually linked to one or more of the other plurality of graphical entity nodes based on relationships between the plurality of entities.
16. The method of claim 1 , wherein at least some of the plurality of graphical entity nodes include interactive elements, which when interacted with by a user, display information regarding the represented entities.
17. A non-transitory computer readable medium storing instructions, execution of which by a computer system, cause the computer system to:
receive metadata from an execution computer system, the metadata generated by the computer system on execution of a workflow including one or more data processing jobs, the metadata including run-time artifacts generated at a time of executing the workflow;
determine a data lineage associated with the workflow based on the metadata;
generate design-time information associated with the workflow based on the data lineage, the design-time information indicative of a design of any of the execution computer system, the workflow, or the one or more data processing jobs included in the workflow; and
generating a data lineage visualization based on the data lineage associated with the workflow; and
causing display of the data lineage visualization.
18. A computer system comprising:
one or more processors; and
a memory having instructions stored thereon, which when executed by the one or more processors, cause the computer system to:
receive metadata from an execution computer system, the metadata generated by the computer system on execution of a workflow including one or more data processing jobs, the metadata including run-time artifacts generated at a time of executing the workflow;
determine a data lineage associated with the workflow based on the metadata;
generate design-time information associated with the workflow based on the data lineage, the design-time information indicative of a design of any of the execution computer system, the workflow, or the one or more data processing jobs included in the workflow; and
generating a data lineage visualization based on the data lineage associated with the workflow; and
causing display of the data lineage visualization.