Systems and methods for tagging datasets using models arranged in a series of nodes
Systems and methods for managing indexing and tagging datasets using a plurality of nodes are disclosed. For example, the system may include one or more memory units storing instructions and one or more processors configured to execute the instructions to perform operations. The operations may include receiving a dataset comprising a plurality of columns and applying a series of nodes to the dataset. Applying the series of nodes may compose applying a first node comprising a machine learning model to generate a first probability, appending a first tag based on the first probability, and selecting second nodes subsequent in the series based on the first probability. Applying the series may include iteratively applying the selected second nodes to generate second probabilities and second tags. The operations may include generating a data structure comprising the first and second probabilities and first and second tags. The operations may include outputting metadata.
1. A system for tagging datasets comprising:
at least one processor; and
at least one non-transitory memory storing instructions that, when executed by the at least one processor cause the at least one processor to perform operations comprising:
receiving a dataset;
inputting the dataset into a first machine learning model;
receiving a first output from the first machine learning model in response to the input dataset, the first output including a first probability of a first category associated with the dataset and a first tag associated with the first category;
selecting, from among a plurality of differently trained second machine learning models, a particular second machine learning model based on the first probability of the first category associated with the dataset, wherein:
the first probability is output by the first machine learning model;
each of the differently trained second machine learning models have been trained to classify data based on a different tag; and
the selected second machine learning model has been further trained to classify data within a plurality of sub-categories of the first category for which the first machine learning model outputs the first probability;
inputting the dataset into the selected second machine learning model after inputting the data set into the first machine learning model;
receiving a second output from the selected second machine learning model based on the dataset, the second output including a second probability of a sub-category of the first category and a second tag associated with the sub-category;
generating a data structure including the first tag and the second tag; and
causing a display to present the generated data structure.
2. The system of claim 1 , wherein the operations further comprise:
training the first machine learning model to classify a column of data within the first category; and
training one of the plurality of the differently trained second machine learning models to classify the column of data within a plurality of sub-categories of the first category.
3. The system of claim 2 , wherein the first machine learning model is configured to generate multiple tags and associated probabilities, wherein each tag and associated probability is associated with a different column of the dataset.
4. The system of claim 1 , wherein the second tag comprises a subcategory of the first tag.
5. The system of claim 1 , wherein the data structure comprises at least one of a tree structure or a vector.
6. The system of claim 1 , wherein the first machine learning model and the selected second machine learning model comprise at least one of a neural network type, a Bayesian network type, a support vector machine type, or a random forest type.
7. The system of claim 1 , wherein the data structure comprises information related to a data schema based on the first and second tags.
8. The system of claim 1 , wherein the first machine learning model and at least one of the plurality of the differently trained second machine learning models are of different machine learning model types.
9. The system of claim 1 , wherein the first tag associated with the first category includes at least one of an address, a name, or a credit card number.
10. The system of claim 1 , the operations further comprising outputting the first and second tags as metadata.
11. The system of claim 1 , wherein selecting the particular second machine learning model from among the plurality of second the differently trained machine learning models includes comprises discarding at least one of the differently trained second machine learning models when the first probability is below a threshold value.
12. The system of claim 1 , the operations further comprising:
selecting, from among a plurality of third machine learning models, a particular third machine learning model based on the second probability;
inputting the dataset into the selected third machine learning model;
receiving a third output from the selected third machine learning model based on the dataset, the third output including a third probability of a child category of the sub-category and a third tag associated with the child category; and
wherein generating the data structure includes the third tag.
13. The system of claim 1 , the operations further comprising:
determining a first vector using the first probability;
determining a second vector using the second probability; and
outputting a distance between a first column and a second column based on the first and second vectors.
14. The system of claim 13 , the operations further comprising causing the display to present a visualization of the distance of the first columns and the second column within a state space.
15. A computer-implemented method for tagging datasets comprising:
receiving a dataset;
inputting the dataset into a first machine learning model;
receiving a first output from the first machine learning model in response to the input dataset, the first output including a first probability of a first category associated with the dataset and a first tag associated with the first category;
selecting, from among a plurality of differently trained second machine learning models, a particular second machine learning model based on the first probability of the first category associated with the dataset, wherein:
the first probability is output by the first machine learning model;
each of the differently trained second machine learning models having been trained to classify data based on a different tag; and
the selected second machine learning model has been further trained to classify data within a plurality of sub-categories of the first category for which the first machine learning model outputs the first probability;
inputting the dataset into the selected second machine learning model after inputting the data set into the first machine learning model;
receiving a second output from the selected second machine learning model based on the dataset, the second output including a second probability of a sub-category of the first category and a second tag associated with the sub-category;
generating a data structure including the first tag and the second tag; and
causing a display to present the generated data structure.
16. The method of claim 15 , further comprising:
training the first machine learning model to classify a column of data as being associated with the first category; and
training one of the plurality of the differently trained second machine learning models to classify the column of data within a plurality of sub-categories of the first category, wherein the training of at least one of the first machine learning model or the one of the differently trained second machine learning models comprises adjusting at least one of weights or activation functions of model nodes to reduce at least one function.
17. The method of claim 15 , wherein the first machine learning model is configured to generate multiple tags and associated probabilities, wherein each tag and associated probability is associated with a different column of the dataset.
18. The method of claim 15 , wherein the first machine learning model and at least one of the plurality of the differently trained second machine learning models are of different machine learning model types.
19. The method of claim 15 , wherein selecting the particular second machine learning model from among the plurality of second machine learning models comprises discarding at least one of the differently trained second machine learning models when the first probability is below a threshold value.
20. The method of claim 15 , further comprising:
selecting, from among a plurality of third machine learning models, a particular third machine learning model based on the second probability;
inputting the dataset into the selected third machine learning model;
receiving a third output from the selected third machine learning model based on the dataset, the third output including a third probability of a child category of the sub-category and a third tag associated with the child category; and
wherein generating the data structure includes the third tag.