Systems and methods for generating data lineage and tracing data changes

1 . A method comprising:

deploying, by a computing device, a tracer agent alongside a data source, the tracer agent configured to monitor the data source for one or more data changes wherein the tracer agent is implemented as a change-event listener executing at the data source;

detecting, by the tracer agent, a data change in a data stream;

introspecting, by the tracer agent, the data change in the data stream;

generating, by the tracer agent, a data structure comprising information about a data element affected by the data change;

receiving the data structure and generating, by the computing device, a data lineage of the data element from the data structure, wherein the data lineage indicates relationships between the data element and one or more other data elements, the one or more other data elements indicated in the received information about the data element;

generating, by the computing device, a lineage template for the data element based on data lineage information as observed by the computing device over a period of time, wherein the lineage template defines a workflow including timing and sequencing for the data element to complete the data change in the data lineage, wherein the lineage template specifies executable timing thresholds and sequence ordering for computer-executable tasks associated with the data element;

generating, by the computing device, a lineage graph based on the data lineage information, wherein the lineage graph tracks the data lineage of the data element;

causing display, by the computing device, of a dashboard of a lineage detection service and, via user-interface controls on the dashboard, receiving input to update an order value and an alert-timeout value in the lineage template;

executing, by the computing device, the workflow according to the lineage template to complete the data change, wherein the executing is performed by a workflow engine that automatically initiates the computer-executable tasks; and

traversing the lineage graph to determine the performance of the workflow, including computing objective performance metrics from timestamps and edge traversal in the lineage graph.

2 . The method of claim 1 , wherein the information about the data element includes identification and versioning information and referentiality information of the data element.

3 . The method of claim 2 , wherein the one or more other data elements being indicated in the referentiality information.

4 . The method of claim 1 , wherein the lineage template defines one or more actions for the workflow, wherein an action of the one or more actions having values that indicate a sequencing of the action, an average time to perform the action, a minimum time to perform the action, a maximum time to perform the action, and a timeout to generate an alert.

5 . The method of claim 1 , wherein the lineage template provides for indicating creation, completion, and status of data tasks.

6 . The method of claim 1 , wherein the lineage graph is updated to show that the workflow is partially completed or completed.

7 . The method of claim 1 , wherein the generating the data lineage includes updating an existing data lineage of the data element.

8 . The method of claim 1 , wherein the generating the lineage template includes identifying an existing lineage template generated for the data element.

9 . A system comprising:

one or more non-transitory machine-readable mediums configured to store instructions; and

one or more processors configured to execute the instructions stored on the one or more non-transitory machine-readable mediums, wherein execution of the instructions causes the one or more processors to carry out a process comprising:

deploying a tracer agent alongside a data source, the tracer agent configured to monitor the data source for one or more data changes, wherein the tracer agent is implemented as a change-event listener executing at the data source;

detecting by the tracer agent a data change in a data stream;

introspecting the data change in the data stream using the tracer agent;

generating, by the tracer agent, a data structure comprising information about a data element affected by the data change;

receiving the data structure and generating a data lineage of the data element, wherein the data lineage indicates relationships between the data element and one or more other data elements, the one or more other data elements indicated in the received information about the data element;

generating a lineage template for the data element based on data lineage information as observed over a period of time, wherein the lineage template defines a workflow including timing and sequencing for the data element to complete the data change in the data lineage, wherein the lineage template specifies executable timing thresholds and sequence ordering for computer-executable tasks associated with the data element;

generating a lineage graph based on the data lineage information, wherein the lineage graph tracks the data lineage of the data element;

causing display, by the computing device, of a dashboard of a lineage detection service and, via user interface controls on the dashboard, receiving input to update an order value and an alert-timeout value in the lineage template;

executing the workflow according to the lineage template to complete the data change, wherein the executing is performed by a workflow engine that automatically initiates the computer-executable tasks; and

traversing the lineage graph to determine the performance of the workflow, including computing objective performance metrics from timestamps and edge traversal in the lineage graph.

10 . The system of claim 9 , wherein the information about the data element includes identification and versioning information and referentiality information of the data element.

11 . The system of claim 10 , wherein the one or more other data elements being indicated in the referentiality information.

12 . The system of claim 9 , wherein the lineage template defines one or more actions for the workflow, wherein an action of the one or more actions having values that indicate a sequencing of the action, an average time to perform the action, a minimum time to perform the action, a maximum time to perform the action, and a timeout to generate an alert.

13 . The system of claim 9 , wherein the lineage graph is updated to show that the workflow is completed.

14 . The system of claim 9 , wherein the lineage graph shows that the workflow is partially completed.

15 . A non-transitory machine-readable medium encoding instructions that when executed by one or more processors cause a process to be carried out, the process including:

deploying a tracer agent alongside a data source, the tracer agent configured to monitor the data source for one or more data changes, wherein the tracer agent is implemented as a change-event listener executing at the data source;

detecting by the tracer agent a data change in a data stream;

introspecting the data change in the data stream using the tracer agent;

generating, by the tracer agent, a data structure comprising information about a data element affected by the data change;

receiving the data structure and generating a data lineage of the data element, wherein the data lineage indicates relationships between the data element and one or more other data elements, the one or more other data elements indicated in the received information about the data element;

generating a lineage template for the data element based on data lineage information as observed over a period of time, wherein the lineage template defines a workflow including timing and sequencing for the data element to complete the data change in the data lineage, wherein the lineage template specifies executable timing thresholds and sequence ordering for computer-executable tasks associated with the data element;

generating a lineage graph based on the data lineage information, wherein the lineage graph tracks the data lineage of the data element;

causing display, by the computing device, of a dashboard of a lineage detection service and, via user-interface controls on the dashboard, receiving input to update an order value and an alert-timeout value in the lineage template;

executing the workflow according to the lineage template to complete the data change, wherein the executing is performed by a workflow engine that automatically initiates the computer-executable tasks; and

traversing the lineage graph to determine the performance of the workflow, including computing objective performance metrics from timestamps and edge traversal in the lineage graph.

16 . The machine-readable medium of claim 15 , wherein the information about the data element includes identification and versioning information and referentiality information of the data element, the one or more other data elements being indicated in the referentiality information.

17 . The machine-readable medium of claim 15 , wherein the lineage template defines one or more actions for the workflow, wherein an action of the one or more actions having values that indicate a sequencing of the action, an average time to perform the action, a minimum time to perform the action, a maximum time to perform the action, and a timeout to generate an alert.