Dataset connector and crawler to identify data lineage and segment data
Systems and methods for connecting datasets are disclosed. For example, a system may include a memory unit storing instructions and a processor configured to execute the instructions to perform operations. The operations may include receiving a plurality of datasets and a request to identify a cluster of connected datasets among the received plurality of datasets. The operations may include selecting a dataset. In some embodiments, the operations include identifying a data schema of the selected dataset and determining a statistical metric of the selected dataset. The operations may include identifying foreign key scores. The operations may include generating a plurality of edges between the datasets based on the foreign key scores, the data schema, and the statistical metric. The operations may include segmenting and returning datasets based on the plurality of edges.
1. A dataset connector system comprising:
one or more memory units storing instructions; and
one or more processors that execute the instructions to perform operations comprising:
receiving, by the dataset connector system, a plurality of datasets;
receiving, by the dataset connector system, a request to identify a cluster of connected datasets among the received plurality of datasets;
selecting, by the dataset connector system, a dataset from among the received plurality of datasets;
identifying, by a data profiling model, a data schema of the selected dataset;
determining, by the data profiling model, a statistical metric of the selected dataset;
identifying, by the data profiling model, a plurality of candidate foreign keys of the selected dataset;
determining, by a data mapping model, respective foreign key scores for individual ones of the plurality of candidate foreign keys;
generating, by the data mapping model, a plurality of edges between the selected dataset and the received plurality of datasets based on the foreign key scores, the data schema, a hierarchical relationship, and the statistical metric;
segmenting, by a data classification model, a cluster of connected datasets comprising the selected dataset, the segmenting based on the plurality of edges, wherein segmenting the datasets comprises:
labelling, by the data classification model, data in the cluster of connected datasets, the labelling indicating that associated data comprises at least one of actual data, synthetic data, or derived data; and
removing, by the data classification model, data from the connected datasets that is labelled as at least one of synthetic data or derived data;
returning, by the dataset connector, the segmented cluster of connected datasets; and
updating at least one of the data profiling model, the data mapping model, or the data classification model using the received plurality of datasets.
2. The dataset connector system of claim 1 , wherein:
identifying the plurality of candidate foreign keys is based on a foreign key index, and
the operations further comprise updating the foreign key index based on one of the respective foreign key scores.
3. The dataset connector system of claim 1 , wherein an individual edge of the plurality of edges comprises an indicator of the hierarchical relationship.
4. The dataset connector system of claim 1 , wherein an individual edge of the plurality of edges comprises an overlap score indicating the amount of overlap between datasets.
5. The dataset connector system of claim 1 , wherein a labelled dataset of the received plurality of datasets comprises synthetic data, and the operations further comprise:
determining, by the data classification model, whether a dataset in the cluster of connected datasets comprises synthetic data based on the labelled dataset.
6. The dataset connector system of claim 5 , the operations further comprising:
retrieving, from a model storage, the data classification model based on a model index and the labelled dataset.
7. The dataset connector system of claim 1 , wherein the data mapping model comprises at least one of a convolutional neural network model or a random forest model.
8. The dataset connector system of claim 1 , wherein the data profiling model comprises at least one of a generative adversarial network model, a recurrent neural network model, a convolutional neural network model, or other machine learning model.
9. The dataset connector system of claim 1 , wherein the data classification model comprises at least one of a convolutional neural network, a random forest model, a recurrent neural network model, a support vector machine model, or another machine learning model.
10. The dataset connector system of claim 1 , the operations further comprising:
determining, by the data connector system, that the selected dataset belongs to a previously segmented cluster of connected datasets, and
wherein segmenting the cluster of connected datasets comprises updating the previously segmented cluster of connected datasets.
11. The dataset connector system of claim 1 , the operations further comprising:
training at least one of the data profiling model, the data mapping model, or the data classification model using a plurality of reference datasets.
12. The dataset connector system of claim 1 , wherein the dataset connector system is hosted on a cloud compute service.
13. The dataset connector system of claim 1 , wherein the respective foreign keys comprise at least one of a phone number, a transaction identifier, an account number, an address, a social security number, a medical record number, or a personal identifier.
14. The dataset connector system of claim 1 , wherein at least one of the data profiling model, the data mapping model, or the data classification model is stored in a model storage.
15. The dataset connector system of claim 1 , the operations further comprising:
retrieving, from a model storage, the data mapping model based on a model index and one of the candidate foreign keys of plurality of candidate foreign keys of the selected keys.
16. The dataset connector system of claim 1 , wherein returning the cluster of connected datasets comprises returning a graphical representation of the cluster of connected datasets.
17. A method for connecting datasets comprising:
receiving, by a dataset connector system, a plurality of datasets;
receiving, by the dataset connector system, a request to identify a cluster of connected datasets among the received plurality of datasets;
selecting, by the dataset connector system, a dataset from among the received plurality of datasets;
identifying, by a data profiling model, a data schema of the selected dataset;
determining, by the data profiling model, a statistical metric of the selected dataset;
identifying, by the data profiling model, a plurality of candidate foreign keys of the selected dataset;
determining, by a data mapping model, respective foreign key scores for individual ones of the plurality of candidate foreign keys;
generating, by the data mapping model, a plurality of edges between the selected dataset and the received plurality of datasets based on the foreign key scores, the data schema, a hierarchical relationship, and the statistical metric;
segmenting, by a data classification model, a cluster of connected datasets comprising the selected dataset, the segmenting based on the plurality of edges, wherein segmenting the datasets comprises:
labelling, by the data classification model, data in the cluster of connected datasets, the labelling indicating that associated data comprises at least one of actual data, synthetic data, or derived data; and
removing, by the data classification model, data from the connected datasets that is labelled as at least one of synthetic data or derived data;
returning, by the dataset connector, the segmented cluster of connected datasets; and
updating at least one of the data profiling model, the data mapping model, or the data classification model using the received plurality of datasets.
18. A method for connecting datasets comprising:
receiving, by a dataset connector system, a plurality of datasets;
receiving, by the dataset connector system, a request to identify a cluster of connected datasets among the received plurality of datasets;
generating, by the dataset connector system, an ephemeral container instance;
selecting, by the ephemeral container instance, a dataset from among the received plurality of datasets;
retrieving, the ephemeral container instance, a data profiling model from a data storage;
identifying, by the data profiling model, a data schema of the selected dataset;
determining, by the data profiling model, a statistical metric of the selected dataset;
identifying, by the data profiling model, a plurality of candidate foreign keys of the selected dataset;
retrieving, by the ephemeral container instance, a data mapping model from a model storage;
determining, by a data mapping model, respective foreign key scores for individual ones of the plurality of candidate foreign keys;
generating, by the data mapping model, a plurality of edges between the selected dataset and the received plurality of datasets based on the foreign key scores, the data schema, a hierarchical relationship and the statistical metric;
retrieving, by the ephemeral container instance, a data classification model from a model storage;
segmenting, by a data classification model, a cluster of connected datasets comprising the selected dataset, the segmenting based on the plurality of edges, wherein segmenting the datasets comprises:
labelling, by the data classification model, data in the cluster of connected datasets, the labelling indicating that associated data comprises at least one of actual data, synthetic data, or derived data; and
removing, by the data classification model, data from the connected datasets that is labelled as at least one of synthetic data or derived data;
returning, by the dataset connector, the segmented cluster of connected datasets;
updating at least one of the data profiling model, the data mapping model, or the data classification model using the received plurality of datasets; and
terminating, by the dataset connector system, the ephemeral container instance.