Managing missing values in datasets for machine learning models
Computer-implemented machines, systems and methods for managing missing values in a dataset for a machine learning model. The method may comprise importing a dataset with missing values; computing data statistics and identifying the missing values; verifying the missing values; updating the missing values; imputing missing values; encoding reasons for why values are missing; combining imputed missing values and the encoded reasons; and recommending models and hyperparameters to handle special or missing values.
1. A computer-implemented method for managing missing values in a dataset for a machine learning model, the method comprising:
identifying a missing value and a corresponding missing value reason for the missing value in an imported dataset comprising a plurality of data point values, the imported dataset being usable for training the machine learning model during a training phase, one or more of the plurality of data point values being associated with one or more features of the machine learning model;
applying an imputation method to generate an imputed feature for the missing value;
replacing the missing value in the dataset with the imputed feature to form a training dataset, the replacing comprising encoding the imputed feature with the corresponding missing value reason for the missing value for which the imputed feature is imputed; and
training the machine learning model using the training dataset.
2. The method of claim 1 , wherein the identifying of the missing value comprises computing data statistics.
3. The method of claim 1 , wherein the imputation method comprises imputing a value of zero for the missing value.
4. The method of claim 1 , wherein the imputation method comprises at least one of a column mean value, a column median value, a column mode value, a column minimum value, or a column maximum value for the missing value.
5. The method of claim 1 , wherein the imputation method comprises use of a deep autoencoder.
6. The method of claim 1 , further comprising utilizing one or more missing or special value handling recommendation modules to recommend the imputation method for imputing the imputed feature of the missing value and an encoding process for providing the reasons for the missing value missing from the dataset.
7. The method of claim 6 , further comprising utilizing an on-line or off-line recommendation module to recommend missing or special value handling methods and hyperparameters, wherein the hyperparameters are used to by the imputation method.
8. A system comprising:
at least one programmable processor; and
a non-transitory machine-readable medium storing instructions that, when executed by the at least one programmable processor, cause the at least one programmable processor to perform operations for managing missing values in a dataset for a machine learning model, the operations comprising:
identifying a missing value and a corresponding missing value reason for the missing value in an imported dataset comprising a plurality of data point values, the imported dataset being usable for training the machine learning model during a training phase, one or more of the plurality of data point values being associated with one or more features of the machine learning model;
applying an imputation method to generate an imputed feature for the missing value;
replacing the missing value in the dataset with the imputed feature to form a training dataset, the replacing comprising encoding the imputed feature with the corresponding missing value reason for the missing value for which the imputed feature is imputed; and
training the machine learning model using the training dataset.
9. The system of claim 8 , wherein the identifying of the missing value comprises computing data statistics.
10. A computer program product comprising a non-transitory machine-readable medium storing instructions that, when executed by at least one programmable processor, cause the at least one programmable processor to perform operations for managing missing values in a dataset for a machine learning model, the operations comprising comprising:
identifying a missing value and a corresponding missing value reason for the missing value in an imported a dataset comprising a plurality of data point values, the imported dataset being usable for training the machine learning model during a training phase, one or more of the plurality of data point values being associated with one or more features of the machine learning model;
applying an imputation method to generate an imputed feature for the missing value;
replacing the missing value in the dataset with the imputed feature to form a training dataset, the replacing comprising encoding the imputed feature with the corresponding missing value reason for the missing value for which the imputed feature is imputed; and
training the machine learning model using the training dataset.
11. The computer program product of claim 10 , wherein the identifying of the missing value comprises computing data statistics.
12. The system of claim 8 , wherein the imputation method comprises imputing a value of zero for the missing value.
13. The system of claim 8 , wherein the imputation method comprises at least one of a column mean value, a column median value, a column mode value, a column minimum value, or a column maximum value for the missing value.
14. The system of claim 8 , wherein the imputation method comprises use of a deep autoencoder.
15. The system of claim 8 , wherein the operations further comprise utilizing one or more missing or special value handling recommendation modules to recommend the imputation method for imputing the imputed feature of the missing value and an encoding process for providing the reasons for the missing value missing from the dataset.
16. The system of claim 15 , wherein the operations further comprise utilizing an on-line or off-line recommendation module to recommend missing or special value handling methods and hyperparameters, wherein the hyperparameters are used to by the imputation method.
17. The computer program product of claim 10 , wherein the imputation method comprises one of imputing a value of zero for the missing value, using a column mean value, using a column median value, using a column mode value, using a column minimum value, using a column maximum value for the missing value, and using a deep autoencoder.
18. The computer program product of claim 10 , wherein the operations further comprise utilizing one or more missing or special value handling recommendation modules to recommend the imputation method for imputing the imputed feature of the missing value and an encoding process for providing the reasons for the missing value missing from the dataset.
19. The computer program product of claim 18 , wherein the operations further comprise utilizing an on-line or off-line recommendation module to recommend missing or special value handling methods and hyperparameters, wherein the hyperparameters are used to by the imputation method.