Machine learning oriented interactive tabular data quality display systems and methods
Certain example embodiments relate to dashboards that help streamline and automate data quality management processes used with machine learning (ML) models and ML-enabled technology. A clean dataset is initialized from a dirty dataset. A search space is the set of all possible combinations of available error detection algorithms and data repair algorithms. A scoring function measures performance of a given error detection algorithm and data repair algorithm combination on the clean dataset. An ML model is trained using the clean dataset. Best error detection and data repair algorithms are selected, based on an optimization on the set of all possible combinations, and the defined scoring function. The selected best error detection algorithm is applied to the clean dataset, and a repaired dataset is generated using the selected best repair algorithm. The clean dataset is set to the repaired dataset. This procedure is repeated until a condition is met.
1 . A machine learning (ML) model dataset preparation method for use with a system having access to a plurality of error detection algorithms, a plurality of data repair algorithms, and an ML model, the method comprising:
(a) accessing a dirty dataset from a data store;
(b) initializing a clean dataset to the accessed dirty dataset;
(c) defining a search space as a set of all possible combinations of the plurality of error detection algorithms and the plurality of data repair algorithms, wherein a scoring function is defined to measure performance of a given error detection algorithm and data repair algorithm combination on the clean dataset;
(d) training the ML model using the clean dataset;
(e) selecting a best error detection algorithm and selecting a best data repair algorithm, based on an optimization on (i) the search space and (ii) the defined scoring function;
(f) applying the selected best error detection algorithm to the clean dataset;
(g) generating a repaired dataset by applying the selected best data repair algorithm to the clean dataset;
(h) setting the clean dataset to the repaired dataset;
(i) repeating (d) through (h) until a condition is met; and
(j) outputting the cleaned dataset.
2 . The method of claim 1 , wherein the training in (d) includes training the ML model using the clean dataset and a plurality of labels.
3 . The method of claim 1 , wherein the scoring function includes a root mean square error (RMSE) calculation and/or a F1 score.
4 . The method of claim 1 , wherein the optimization is Bayesian optimization.
5 . The method of claim 1 , wherein (d) through (h) are performable up to a predetermined number of repetitions.
6 . The method of claim 5 , further comprising computing accuracy of the trained ML model with each repetition following (d) and prior to (i).
7 . The method of claim 6 , wherein further repetitions cease provided that the computed accuracy exceeds a predetermined threshold.
8 . The method of claim 1 , wherein the cleaned dataset output in (j) is used as training data.
9 . The method of claim 1 , wherein the dirty dataset is accessed from a file and/or a database.
10 . The method of claim 1 , wherein the plurality of error detection algorithms and the plurality of data repair algorithms are implemented in software modules external to the system.
11 . The method of claim 10 , wherein the software modules are accessible by API calls.
12 . The method of claim 1 , further comprising receiving a user selection, wherein the set of all possible combinations of the plurality of error detection algorithms and the plurality of data repair algorithms is limited by the received user selection.
13 . The method of claim 1 , further comprising storing parameters of the selected best error detection algorithm and the selected best data repair algorithm associated with each respective repetition.
14 . The method of claim 13 , further comprising storing a representation of the repaired dataset associated with each respective repetition.
15 . The method of claim 1 , wherein the repetition includes iteratively cleaning the dirty dataset from the data store, and further comprising, for each repetition:
storing to a structured data type information about the selected best error detection algorithm and the selected best data repair algorithm for the respective repetition, including an identification of the selected best error detection algorithm and the selected best data repair algorithm for the respective repetition and used parameters and artifacts.
16 . The method of claim 1 , wherein the error detection algorithms comprise automatically generated error detection rules and user-defined error detection rules.
17 . The method of claim 1 , further comprising enabling a user to tag dirty values.
18 . The method of claim 1 , wherein at least some of the plurality of error detection algorithms are run sequentially within a single repetition of (d) through (h), prior to one of the plurality of data repair algorithms being implemented.
19 . The method of claim 18 , wherein application of the error detection algorithms generates a plurality of detected errors, the method further comprising removing redundancies from the detected errors.
20 . A non-transitory computer readable storage medium storing instructions that, when executed, cause a processor of a system having access to a plurality of error detection algorithms, a plurality of data repair algorithms, and an ML model, to perform operations comprising:
(a) accessing a dirty dataset from a data store;
(b) initializing a clean dataset to the accessed dirty dataset;
(c) defining a search space as a set of all possible combinations of the plurality of error detection algorithms and the plurality of data repair algorithms, wherein a scoring function is defined to measure performance of a given error detection algorithm and data repair algorithm combination on the clean dataset;
(d) training the ML model using the clean dataset;
(e) selecting a best error detection algorithm and selecting a best data repair algorithm, based on an optimization on (i) the search space and (ii) the defined scoring function;
(f) applying the selected best error detection algorithm to the clean dataset;
(g) generating a repaired dataset by applying the selected best data repair algorithm to the clean dataset;
(h) setting the clean dataset to the repaired dataset;
(i) repeating (d) through (h) until a condition is met; and
(j) outputting the cleaned dataset.
21 . The non-transitory computer readable storage medium of claim 20 , wherein the optimization is Bayesian optimization.
22 . The non-transitory computer readable storage medium of claim 20 , wherein:
(d) through (h) are performable up to a predetermined number of repetitions;
accuracy of the trained ML model is computed with each repetition following (d) and prior to (i); and
further repetitions cease provided that the computed accuracy exceeds a predetermined threshold.
23 . The non-transitory computer readable storage medium of claim 20 , wherein the set of all possible combinations of the plurality of error detection algorithms and the plurality of data repair algorithms is limited based on a user selection of at least one error detection algorithm.
24 . The non-transitory computer readable storage medium of claim 20 , wherein the repetition includes iteratively cleaning the dirty dataset from the data store, and
for each repetition: information about the selected best error detection algorithm and the selected best data repair algorithm for the respective repetition is stored to a structured data type, the information including an identification of the selected best error detection algorithm and the selected best data repair algorithm for the respective repetition and used parameters and artifacts.
25 . A machine learning (ML) model dataset preparation system having access to a plurality of error detection algorithms, a plurality of data repair algorithms, and an ML model, the system comprising:
a data store storing a dirty dataset;
a memory and at least one processor configured to execute instructions stored in the memory to perform operations comprising:
(a) initializing a clean dataset to the dirty dataset;
(b) defining a search space as a set of all possible combinations of the plurality of error detection algorithms and the plurality of data repair algorithms, wherein a scoring function is defined to measure performance of a given error detection algorithm and data repair algorithm combination on the clean dataset;
(c) training the ML model using the clean dataset;
(d) selecting a best error detection algorithm and selecting a best data repair algorithm, based on an optimization on (i) the search space and (ii) the defined scoring function;
(e) applying the selected best error detection algorithm to the clean dataset;
(g) generating a repaired dataset by applying the selected best data repair algorithm to the clean dataset;
(g) setting the clean dataset to the repaired dataset;
(h) repeating (c) through (g) until a condition is met; and
(i) outputting the cleaned dataset.
26 . The system of claim 25 , wherein the optimization is Bayesian optimization.
27 . The system of claim 25 , wherein:
(c) through (g) are performable up to a predetermined number of repetitions;
accuracy of the trained ML model is computed with each repetition following (c) and prior to (h); and
further repetitions cease provided that the computed accuracy exceeds a predetermined threshold.
28 . The system of claim 25 , wherein the set of all possible combinations of the plurality of error detection algorithms and the plurality of data repair algorithms is limited based on a user selection of at least one error detection algorithm.
29 . The system of claim 25 , wherein the repetition includes iteratively cleaning the dirty dataset from the data store, and
for each repetition: information about the selected best error detection algorithm and the selected best data repair algorithm for the respective repetition is stored to a structured data type, the information including an identification of the selected best error detection algorithm and the selected best data repair algorithm for the respective repetition and used parameters and artifacts.