Systems and methods for predicting correct or missing data and data anomalies
The present disclosure is directed to systems and methods for predicting and correcting data anomalies. In one example aspect, data is received by the system. The system may analyze the data by profiling the data for certain profiling statistics (e.g., min, max, mean, cardinality, etc.). At least one machine-learning algorithm (e.g., a Random-Forest algorithm) may be applied to the profiled data to identify potential relationships among certain data columns in the data. Once certain relationships are identified, the data that is related may be extracted to form an itemset. A second machine-learning algorithm (e.g., Frequent Pattern Growth algorithm) may be applied to the itemset to identify certain frequencies of related values in the itemset. Low frequency values may indicate anomalies in the dataset. If an anomaly is detected, the system may be configured to provide an intelligent remedial action, such as substituting certain values and/or filling in a missing value.
1. A method for predicting data quality in a dataset, comprising:
receiving a first dataset;
applying a first machine-learning algorithm to the first dataset, wherein the first machine-learning algorithm identifies at least one relationship between a first data column and a second data column in the first dataset;
based on the identification of the at least one relationship between the first data column and the second data column in the first dataset, generating a second dataset, wherein the second dataset is a subset of the first dataset;
concatenating a plurality of column headers in the second dataset;
generating an itemset based on the concatenation of the plurality of column headers in the second dataset; and
applying a second machine-learning algorithm to the itemset, wherein the second machine-learning algorithm identifies at least one frequency value associated with the at least one relationship between the first data column and the second data column in the first dataset.
2. The method of claim 1 , further comprising identifying at least one anomaly in the itemset by identifying at least one missing value in the itemset.
3. The method of claim 2 , further comprising generating at least one suggestion for remediating the at least one anomaly including at least one suggested value to fill in the at least one missing value in the itemset.
4. The method of claim 3 , wherein the at least one suggested value is at least one of: zero, a mean value, a min value, and a max value.
5. The method of claim 1 , wherein the first machine-learning algorithm is a random-forest algorithm.
6. The method of claim 1 , wherein the second machine-learning algorithm is a frequent pattern growth algorithm.
7. The method of claim 3 , wherein the at least one anomaly in the itemset is a relationship anomaly in at least one data record.
8. The method of claim 7 , wherein the at least one suggestion is a suggestion for correcting the relationship anomaly in the at least one data record.
9. The method of claim 8 , wherein the suggestion for correcting the relationship anomaly comprises at least one substitute value.
10. The method of claim 3 , further comprising:
receiving at least one user input accepting the at least one suggestion; and
based on the at least one user input, applying the at least one suggestion to the itemset.
11. The method of claim 1 , wherein the first dataset and the second dataset are at least one of: a datafile, a data lake, a flat file, and a data stream.
12. A system for correcting data anomalies in a dataset, comprising:
a memory configured to store non-transitory computer readable instructions; and
a processor communicatively coupled to the memory, wherein the processor, when executing the non-transitory computer readable instructions, is configured to:
receive a dataset;
apply a first machine-learning algorithm to the dataset, wherein the first machine-learning algorithm identifies at least one relationship between a first data column and a second data column in the dataset;
based on the identification of the at least one relationship between the first data column and the second data column in the dataset, generate an itemset; and
apply a second machine-learning algorithm to the itemset, wherein the second machine-learning algorithm identifies at least one frequency value associated with the at least one relationship between the first data column and the second data column in the first dataset.
13. The system of claim 12 , further comprising identifying at least one anomaly in the at least one data record including a relationship anomaly between at least one data value in the first data column and at least one data value in the second data column.
14. The system of claim 13 , wherein the relationship anomaly is associated with a geographic location.
15. The system of claim 13 , wherein the relationship anomaly is associated with an asset symbol, wherein the asset symbol is at least one of: a stock ticker and a cryptocurrency ticker.
16. The system of claim 12 , wherein the first machine-learning algorithm is a random-forest algorithm.
17. The system of claim 12 , wherein the second machine-learning algorithm is a frequent pattern growth algorithm.
18. The system of claim 13 , wherein at least one anomaly is at least one of: a missing value, an incorrect value, and a formatting error.
19. The system of claim 13 , further configured to generate at least one output table displaying the at least one anomaly in the at least one data record in the itemset and a corrected version of the at least one data record without the at least one anomaly.
20. A computer-readable media storing non-transitory computer executable instructions that when executed cause a computing system to perform a method for predicting and correcting data anomalies comprising:
receiving a first dataset;
comparing a first data column in the first dataset to a plurality of other data columns in the first dataset;
based on the comparison results of the first data column and the plurality of other data columns, identifying a first relationship between the first data column and a second data column in the first dataset;
applying a first machine-learning algorithm to the first data column and the second data column, wherein the first machine-learning algorithm identifies a second relationship between the first data column and the second data column in the first dataset;
based on the identification of the second relationship between the first data column and the second data column in the first dataset, generating a second dataset, wherein the second dataset is a subset of the first dataset;
concatenating a plurality of column headers in the second dataset;
generating an itemset based on the concatenation of the plurality of column headers in the second dataset; and
applying a second machine-learning algorithm to the itemset, wherein the second machine-learning algorithm identifies at least one frequency value associated with the second relationship between the first data column and the second data column in the first dataset.