Data lineage tracking
The present application discloses a method, system, and computer system for managing lineage data for data entities. The method includes generating lineage data, wherein generating the lineage data, and storing and indexing, in a data structure, the lineage data in association with the selected data entity. The generating the lineage data includes selecting a selected data entity, obtaining a query tree that was used to generate the selected data entity, and determining lineage data for the selected data entity based at least in part on the query tree.
1. A method comprising:
executing a job on one or more workers of a compute resource, wherein executing the one or more jobs further comprises invoking one or more data entities;
detecting that a data entity in the one or more data entities is corrupt in response to determination that execution of the job has failed;
identifying a lineage data identifier associated with the data entity based on a mapping of lineage data identifiers to data entity identifiers;
accessing lineage data that is stored in association with the identified lineage data identifier, the lineage data having been generated based on a query tree that was used to generate the data entity, and the lineage data identifying a set of data entities that rely on the data entity;
identifying, based on the lineage data, one or more upstream data entities from the data entity;
determining that the one or more upstream data entities from which the data entity depends from has been corrupted; and
providing an indication of the corruption in the one or more upstream data entities or the data entity to a client device.
2. The method of claim 1 , further comprising:
in response to determining that the data entity is impacted by the corruption in the one or more upstream data entities, causing a corrective action to be performed.
3. The method of claim 2 , wherein causing the corrective action to be performed comprises:
replacing the data entity.
4. The method of claim 2 , wherein causing the corrective action to be performed comprises:
recalculating the data entity based on an updated version of the one or more upstream data entities.
5. The method of claim 1 , wherein the data entity is a table or a column of a table.
6. The method of claim 1 , further comprising:
obtaining a query that was used to create the data entity;
parsing the query to generate the query tree;
generating the data lineage for the data entity based on the query tree; and
storing the data lineage in association with the data entity.
7. The method of claim 6 , wherein generating the lineage data comprises:
traversing the query tree and extracting a set of lineage data for one or more data entities that are created in the query tree.
8. A system comprising:
one or more computer processors; and
one or more computer-readable mediums storing instructions that, when executed by the one or more computer processors, cause the system to perform operations comprising:
executing a job on one or more workers of a compute resource, wherein executing the one or more jobs further comprises invoking one or more data entities;
detecting that a data entity in the one or more data entities is corrupt in response to determination that execution of the job has failed;
identifying a lineage data identifier associated with the data entity based on a mapping of lineage data identifiers to data entity identifiers;
accessing lineage data that is stored in association with the identified lineage data identifier, the lineage data having been generated based on a query tree that was used to generate the data entity, and the lineage data identifying a set of data entities that rely on the data entity;
identifying, based on the lineage data, one or more upstream data entities from the data entity;
determining that the one or more upstream data entities from which the data entity depends from has been corrupted; and
providing an indication of the corruption in the one or more upstream data entities or the data entity to a client device.
9. The system of claim 8 , the operations further comprising:
in response to determining that the data entity is impacted by the corruption in the one or more upstream data entities, causing a corrective action to be performed.
10. The system of claim 9 , wherein causing the corrective action to be performed comprises:
replacing the data entity.
11. The system of claim 9 , wherein causing the corrective action to be performed comprises:
recalculating the data entity based on an updated version of the one or more upstream data entities.
12. The system of claim 8 , wherein the data entity is a table or a column of a table.
13. The system of claim 8 , the operations further comprising:
obtaining a query that was used to create the data entity;
parsing the query to generate the query tree;
generating the data lineage for the data entity based on the query tree; and
storing the data lineage in association with the data entity.
14. The system of claim 13 , wherein generating the lineage data comprises:
traversing the query tree and extracting a set of lineage data for one or more data entities that are created in the query tree.
15. A non-transitory computer-readable medium storing instructions that, when executed by one or more computer processors of one or more computing devices, cause the one or more computing devices to perform operations comprising:
executing a job on one or more workers of a compute resource, wherein executing the one or more jobs further comprises invoking one or more data entities;
detecting that a data entity in the one or more data entities is corrupt in response to determination that execution of the job has failed;
identifying a lineage data identifier associated with the data entity based on a mapping of lineage data identifiers to data entity identifiers;
accessing lineage data that is stored in association with the identified lineage data identifier, the lineage data having been generated based on a query tree that was used to generate the data entity, and the lineage data identifying a set of data entities that rely on the data entity;
identifying, based on the lineage data, one or more upstream data entities from the data entity;
determining that the one or more upstream data entities from which the data entity depends from has been corrupted; and
providing an indication of the corruption in the one or more upstream data entities or the data entity to a client device.
16. The non-transitory computer-readable medium of claim 15 , the operations further comprising:
in response to determining that the data entity is impacted by the corruption in the one or more upstream data entities, causing a corrective action to be performed.
17. The non-transitory computer-readable medium of claim 16 , wherein causing the corrective action to be performed comprises:
replacing the data entity.
18. The non-transitory computer-readable medium of claim 16 , wherein causing the corrective action to be performed comprises:
recalculating the data entity based on an updated version of the one or more upstream data entities.
19. The non-transitory computer-readable medium of claim 15 , wherein the data entity is a table or a column of a table.
20. The non-transitory computer-readable medium of claim 15 , the operations further comprising:
obtaining a query that was used to create the data entity;
parsing the query to generate the query tree;
generating the data lineage for the data entity based on the query tree by traversing the query tree and extracting a set of lineage data for one or more data entities that are created in the query tree; and
storing the data lineage in association with the data entity.