Natural language based data lineage tracking in complex data ecosystems
Various embodiments of the present disclosure provide multi-stage data lineage tracking techniques for automatically generating holistic and accurate data catalog for complex data ecosystems. The techniques may include generating a critical attribute collection for a data-related task based on one or more natural language descriptions for the data-related task. The techniques include receiving a data lineage map that defines a plurality of hierarchical data layers for a data ecosystem associated with the data-related task. The techniques include generating a critical attribute map for the data-related task by identifying a data element for a critical attribute at each of the plurality of hierarchical data layers of the data lineage map. The techniques include identifying noncritical attributes for the data related task generating a task attribute map for the data-related task based on the noncritical attributes and the critical attribute map.
1. A computer-implemented method, comprising:
generating, by one or more processors and based on processing one or more natural language descriptions for a data-related task and using an attribute selection model, a critical attribute collection for the data-related task;
receiving, by the one or more processors, a data lineage map that defines a plurality of hierarchical data layers for a data ecosystem associated with the data-related task;
generating, by the one or more processors, a critical attribute map for the data-related task by identifying a data element for a critical attribute from the critical attribute collection at each of the plurality of hierarchical data layers of the data lineage map;
identifying, by the one or more processors, a plurality of noncritical attributes related to the critical attribute collection for the data-related task based on the critical attribute collection;
generating, by the one or more processors, an augmented data entity by augmenting, using the critical attribute map and the plurality of noncritical attributes, a data entity comprising the data lineage map;
generating, by the one or more processors and based on the augmented data entity, a task attribute map for the data-related task that is configured to depict attribute relationships across the plurality of hierarchical data layers; and
responsive to detecting a changed data element corresponding to a particular critical attribute at a hierarchical data layer of the plurality of hierarchical data layers, automatically updating, by the one or more processors and based on the task attribute map for the data-related task, one or more corresponding data elements at one or more additional hierarchical data layers of the plurality of hierarchical data layers.
2. The computer-implemented method of claim 1 , wherein:
(i) the critical attribute map comprises a plurality of source-target layer relationships for the particular critical attribute, and
(ii) a source-target layer relationship of the plurality of source-target layer relationships is indicative of a mapping between a source data element within a first hierarchical data layer of the plurality of hierarchical data layers and a target data element within a second hierarchical data layer of the plurality of hierarchical data layers.
3. The computer-implemented method of claim 2 , wherein the source-target layer relationship is based on a comparison between source metadata corresponding to the source data element and target metadata corresponding to the target data element.
4. The computer-implemented method of claim 3 , further comprising generating the source-target layer relationship, wherein generating the source-target layer relationship comprises:
receiving the source metadata and the target metadata;
generating, using a relationship mapping model, a semantic similarity score based on the source metadata and the target metadata; and
generating an initial source-target layer relationship based on the semantic similarity score.
5. The computer-implemented method of claim 4 , wherein generating the source-target layer relationship further comprises:
generating, using the relationship mapping model, initial validation data for the initial source-target layer relationship based on query logic indicative of a plurality of data queries between the first hierarchical data layer and the second hierarchical data layer; and
generating the source-target layer relationship based on the initial validation data.
6. The computer-implemented method of claim 5 , wherein generating the source-target layer relationship further comprises:
generating, using the relationship mapping model, final validation data for the initial source-target layer relationship based on the initial validation data; and
generating the source-target layer relationship based on the final validation data.
7. The computer-implemented method of claim 1 , wherein the critical attribute collection comprises a subset of critical attributes from a plurality of attributes accessible from the data ecosystem.
8. The computer-implemented method of claim 1 , wherein the plurality of hierarchical data layers comprises one or more hierarchical data layers between (i) an originating layer comprising one or more first data sources and (ii) a consumption layer comprising one or more second data sources.
9. The computer-implemented method of claim 8 , wherein generating the critical attribute map for the data-related task comprises iteratively generating a respective source-target layer relationship for the critical attribute starting with the consumption layer and ending with the originating layer.
10. The computer-implemented method of claim 1 , wherein the one or more natural language descriptions comprise a key performance indicator task description, and wherein generating the critical attribute collection comprises:
generating, using the attribute selection model, a plurality of candidate data entities from the key performance indicator task description;
generating, using the attribute selection model, a plurality of semantic similarity scores between the plurality of candidate data entities and a plurality of data attributes from the data ecosystem; and
generating, using the attribute selection model, the critical attribute collection based on the plurality of semantic similarity scores and the plurality of data attributes from the data ecosystem.
11. A system comprising one or more processors and at least one memory storing processor-executable instructions that, when executed by any one or more of the one or more processors, cause the one or more processors to perform operations comprising:
generating, based on processing one or more natural language descriptions for a data-related task and using an attribute selection model, a critical attribute collection for the data-related task;
receiving a data lineage map that defines a plurality of hierarchical data layers for a data ecosystem associated with the data-related task;
generating a critical attribute map for the data-related task by identifying a data element for a critical attribute from the critical attribute collection at each of the plurality of hierarchical data layers of the data lineage map;
identifying a plurality of noncritical attributes related to the critical attribute collection for the data-related task based on the critical attribute collection;
generating an augmented data entity by augmenting, using the critical attribute map and the plurality of noncritical attributes, a data entity comprising the data lineage map;
generating, based on the augmented data entity, a task attribute map for the data-related task that is configured to depict attribute relationships across the plurality of hierarchical data layers; and
responsive to detecting a changed data element corresponding to a particular critical attribute at a hierarchical data layer of the plurality of hierarchical data layers, automatically updating, based on the task attribute map for the data-related task, one or more corresponding data elements at one or more additional hierarchical data layers of the plurality of hierarchical data layers.
12. The system of claim 11 , wherein:
(i) the critical attribute map comprises a plurality of source-target layer relationships for the particular critical attribute, and
(ii) a source-target layer relationship of the plurality of source-target layer relationships is indicative of a mapping between a source data element within a first hierarchical data layer of the plurality of hierarchical data layers and a target data element within a second hierarchical data layer of the plurality of hierarchical data layers.
13. The system of claim 12 , wherein the source-target layer relationship is based on a comparison between source metadata corresponding to the source data element and target metadata corresponding to the target data element.
14. The system of claim 13 , the operations further comprising generating the source-target layer relationship, wherein generating the source-target layer relationship comprises:
receiving the source metadata and the target metadata;
generating, using a relationship mapping model, a semantic similarity score based on the source metadata and the target metadata; and
generating an initial source-target layer relationship based on the semantic similarity score.
15. The system of claim 14 , wherein generating the source-target layer relationship further comprises:
generating, using the relationship mapping model, initial validation data for the initial source-target layer relationship based on query logic indicative of a plurality of data queries between the first hierarchical data layer and the second hierarchical data layer; and
generating the source-target layer relationship based on the initial validation data.
16. The system of claim 15 , wherein generating the source-target layer relationship further comprises:
generating, using the relationship mapping model, final validation data for the initial source-target layer relationship based on the initial validation data; and
generating the source-target layer relationship based on the final validation data.
17. One or more non-transitory computer-readable storage media storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:
generating, based on processing one or more natural language descriptions for a data-related task and using an attribute selection model, a critical attribute collection for the data-related task;
receiving a data lineage map that defines a plurality of hierarchical data layers for a data ecosystem associated with the data-related task;
generating a critical attribute map for the data-related task by identifying a data element for a critical attribute from the critical attribute collection at each of the plurality of hierarchical data layers of the data lineage map;
identifying a plurality of noncritical attributes related to the critical attribute collection for the data-related task based on the critical attribute collection;
generating, by the one or more processors, an augmented data entity by augmenting, using the critical attribute map and the plurality of noncritical attributes, a data entity comprising the data lineage map;
generating, based on the augmented data entity, a task attribute map for the data-related task that is configured to depict attribute relationships across the plurality of hierarchical data layers; and
responsive to detecting a changed data element corresponding to a particular critical attribute at a hierarchical data layer of the plurality of hierarchical data layers, automatically updating, based on the task attribute map for the data-related task, one or more corresponding data elements at one or more additional hierarchical data layers of the plurality of hierarchical data layers.
18. The one or more non-transitory computer-readable storage media of claim 17 , wherein the critical attribute collection comprises a subset of critical attributes from a plurality of attributes accessible from the data ecosystem.
19. The one or more non-transitory computer-readable storage media of claim 17 , wherein the plurality of hierarchical data layers comprises one or more hierarchical data layers between (i) an originating layer comprising one or more first data sources and (ii) a consumption layer comprising one or more second data sources.
20. The one or more non-transitory computer-readable storage media of claim 19 , wherein generating the critical attribute map for the data-related task comprises iteratively generating a respective source-target layer relationship for the critical attribute starting with the consumption layer and ending with the originating layer.