Informed labeling of records for faster discovery of business critical information
One example method includes crawling data included in a dataset, based on the crawling of the data, creating and/or obtaining metadata concerning the data, crawling the metadata and obtaining information about a data change event involving the data, based on the crawling of the metadata, creating a meta label that documents occurrence of the data change event, and associating the meta label with the metadata. An analysis of the data, metadata, and meta label, may be performed and a trigger generated based on the analysis.
1. A method, comprising:
crawling, by a data crawler, data included in a dataset;
based on the crawling of the data, creating and/or obtaining, by the data crawler, metadata concerning the data, the metadata including new metadata created by the data crawler, as well as previously obtained metadata that is associated with the data and that was obtained by a prior crawling of the data;
creating or modifying, by the data crawler, a data lineage record for the data by linking the previously obtained metadata with the new metadata;
crawling, by a meta crawler, the metadata and obtaining information about a data change event involving the data, wherein the meta crawler is operable to examine two or more directly linked pieces of metadata parsed from a common record and, based on this, calculate or otherwise determine any one or more of: how much data has been added to, or removed from, the record; whether the nature of the content within the record has changed, and the type of the change that has occurred; and, if sensitive data has been added to, or removed from, the record;
based on the crawling of the metadata, creating, by the meta crawler, a meta label that documents occurrence of the data change event, wherein the meta label indicates a state of the data at a time the data was crawled; and
associating the meta label with the metadata.
2. The method as recited in claim 1 , further comprising analyzing one or more of the data, metadata, and meta label.
3. The method as recited in claim 2 , further comprising, based on the analyzing, generating an alert concerning the data change event, and transmitting the alert to an entity for action.
4. The method as recited in claim 3 , wherein the alert includes information indicating why the alert was generated.
5. The method as recited in claim 1 , wherein crawling of the data is triggered by occurrence of an event affecting the data.
6. The method as recited in claim 1 , wherein crawling of the metadata is triggered by crawling of the data.
7. The method as recited in claim 1 , wherein the data is a part of contents of a file, and the metadata is created based on the contents and the data change event.
8. The method as recited in claim 1 , wherein the data change event is automatically detected based on both the metadata, and metadata concerning an earlier data change event that took place prior to the data change event.
9. The method as recited in claim 1 , wherein the data that is crawled is distributed across multiple different locations.
10. A non-transitory computer readable storage medium having stored therein instructions that are executable by one or more hardware processors to perform operations comprising:
crawling, by a data crawler, data included in a dataset;
based on the crawling of the data, creating and/or obtaining, by the data crawler, metadata concerning the data, the metadata including new metadata created by the data crawler, as well as previously obtained metadata that is associated with the data and that was obtained by a prior crawling of the data;
creating or modifying, by the data crawler, a data lineage record for the data by linking the previously obtained metadata with the new metadata;
crawling, by a meta crawler, the metadata and obtaining information about a data change event involving the data, wherein the meta crawler is operable to examine two or more directly linked pieces of metadata parsed from a common record and, based on this, calculate or otherwise determine any one or more of: how much data has been added to, or removed from, the record; whether the nature of the content within the record has changed, and the type of the change that has occurred; and, if sensitive data has been added to, or removed from, the record;
based on the crawling of the metadata, creating, by the meta crawler, a meta label that documents occurrence of the data change event, wherein the meta label indicates a state of the data at a time the data was crawled; and
associating the meta label with the metadata.
11. The non-transitory computer readable storage medium as recited in claim 10 , wherein the operations further comprise analyzing one or more of the data, metadata, and meta label.
12. The non-transitory computer readable storage medium as recited in claim 11 , wherein the operations further comprise, based on the analyzing, generating an alert concerning the data change event, and transmitting the alert to an entity for action.
13. The non-transitory computer readable storage medium as recited in claim 12 , wherein the alert includes information indicating why the alert was generated.
14. The non-transitory computer readable storage medium as recited in claim 10 , wherein crawling of the data is triggered by occurrence of an event affecting the data.
15. The non-transitory computer readable storage medium as recited in claim 10 , wherein crawling of the metadata is triggered by crawling of the data.
16. The non-transitory computer readable storage medium as recited in claim 10 , wherein the data is a part of contents of a file, and the metadata is created based on the contents and the data change event.
17. The non-transitory computer readable storage medium as recited in claim 10 , wherein the data change event is automatically detected based on both the metadata, and metadata concerning an earlier data change event that took place prior to the data change event.
18. The non-transitory computer readable storage medium as recited in claim 10 , wherein the data that is crawled is distributed across multiple different locations.