Systems and methods for generating data lineage
Systems and methods for lineage data tracking are disclosed. Domain-specific code is obtained in response to a build request and the domain-specific code is converted to process code. The domain-specific code includes an instance of a domain-specific language configured for a specific domain. The process code includes at least one breadcrumb instrumentation element automatically inserted during the conversion process by a conversion engine and the breadcrumb instrumentation element is not defined in the domain-specific code. The process code is executed including execution of one or more activities defined by the domain-specific code and the at least one breadcrumb instrumentation element. During execution of the at least one breadcrumb instrumentation element, a lineage data structure is generated. The lineage data structure is configured to store data elements representative of a mapping between an input data element and an output data element. The lineage data structure is stored in a data store.
1 . A system, comprising:
a non-transitory memory;
a processor communicatively coupled to the non-transitory memory, wherein the processor is configured to read a set of instructions to:
obtain domain-specific code in response to a build request, wherein the domain-specific code includes an instance of a domain-specific language configured for a specific domain;
generate process code, including (i) converting the domain-specific code to a format of the process code and (ii) automatically inserting at least one breadcrumb instrumentation element, wherein the at least one breadcrumb instrumentation element is automatically inserted during conversion by a conversion engine, and
wherein the breadcrumb instrumentation element is not defined in the domain-specific code;
execute the process code, wherein execution of the process code includes;
execution of one or more activities defined by the domain-specific code; and
generation of a lineage data structure, via execution of the at least one breadcrumb instrumentation element, that stores data elements representative of a mapping between an input data element and an output data element;
store the lineage data structure in a data store; and
implement a data lineage review process,
wherein the at least one breadcrumb instrumentation element is inserted in the process code at a location corresponding to a function type defined in the domain-specific code.
2 . The system of claim 1 , wherein the lineage data structure comprises a tuple defining a set of inputs, a set of outputs, and a corresponding activity of the one or more activities defined by the domain-specific code.
3 . The system of claim 2 , wherein the set of inputs and the set of outputs each comprise at least one instance comprising a type and an identifier.
4 . The system of claim 1 , wherein the lineage data structure comprise a symbolic data element defining a set of inputs and a corresponding activity of the one or more activities defined by the domain-specific code.
5 . The system of claim 1 , wherein the at least one breadcrumb instrumentation element comprises a function call to a function defined by a processing engine configured to execute the process code.
6 . The system of claim 1 , wherein the breadcrumb instrumentation element comprises a scoping mechanism.
7 . The system of claim 6 , wherein the scoping mechanism is configured to mark a stack of potential inputs, and wherein the lineage data structure comprises inputs occurring after the mark in the stack.
8 . The system of claim 1 , wherein the process code comprises Java™ code.
9 . A computer-implemented method, comprising:
obtaining domain-specific code in response to a build request, wherein the domain-specific code includes an instance of a domain-specific language configured for a specific domain;
generating process code, including (i) converting the domain-specific code to a format of the process code and (ii automatically inserting at least one breadcrumb instrumentation element, wherein the at least one breadcrumb instrumentation element is automatically inserted during conversion by a conversion engine, and
wherein the breadcrumb instrumentation element is not defined in the domain-specific code;
executing the process code, wherein execution of the process code includes:
execution of one or more activities defined by the domain-specific code; and
generation of a lineage data structure, via execution of the at least one breadcrumb instrumentation element, that stores data elements representative of a mapping between an input data element and an output data element;
storing the lineage data structure in a data store; and
implementing a data lineage review process,
wherein the at least one breadcrumb instrumentation element is inserted in the process code at a location corresponding to a function type defined in the domain-specific code.
10 . The computer-implemented method of claim 9 , wherein the lineage data structure comprises a tuple defining a set of inputs, a set of outputs, and a corresponding activity of the one or more activities defined by the domain-specific code.
11 . The computer-implemented method of claim 9 , wherein the lineage data structure comprise a symbolic data element defining a set of inputs and a corresponding activity of the one or more activities defined by the domain-specific code.
12 . The computer-implemented method of claim 9 , wherein the at least one breadcrumb instrumentation element comprises a function call to a function defined by a processing engine configured to execute the process code.
13 . The computer-implemented method of claim 9 , wherein the breadcrumb instrumentation element comprises a scoping mechanism.
14 . The computer-implemented method of claim 13 , wherein the scoping mechanism is configured to mark a stack of potential inputs, and wherein the lineage data structure comprises inputs occurring after the mark in the stack.
15 . A non-transitory computer readable medium having instructions stored thereon, wherein the instructions, when executed by at least one processor, cause at least one device to perform operations comprising:
obtaining domain-specific code in response to a build request, wherein the domain-specific code includes an instance of a domain-specific language configured for a specific domain;
generating process code, including (i) converting the domain-specific code to a format of the process code and (ii) automatically inserting at least one breadcrumb instrumentation element, wherein the at least one breadcrumb instrumentation element is automatically inserted during conversion by a conversion engine, and
wherein the breadcrumb instrumentation element is not defined in the domain-specific code;
executing the process code, wherein execution of the process code includes:
execution of one or more activities defined by the domain-specific code; and
generation of a lineage data structure, via execution of the at least one breadcrumb instrumentation element, that stores data elements representative of a mapping between an input data element and an output data element;
storing the lineage data structure in a data store; and
implementing a data lineage review process,
wherein the lineage data structure comprises a tuple defining a set of inputs, a set of outputs, and a corresponding activity of the one or more activities defined by the domain-specific code.
16 . The non-transitory computer readable medium of claim 15 , wherein the lineage data structure comprise a symbolic data element defining a set of inputs and a corresponding activity of the one or more activities defined by the domain-specific code.
17 . The non-transitory computer readable medium of claim 15 , wherein the at least one breadcrumb instrumentation element comprises a function call to a function defined by a processing engine configured to execute the process code.