Managing missing values in datasets for machine learning models
Computer-implemented machines, systems and methods for managing missing values in a dataset for a machine learning model. The method may comprise importing a dataset with missing values; computing data statistics and identifying the missing values; verifying the missing values; updating the missing values; imputing missing values; encoding reasons for why values are missing; combining imputed missing values and the encoded reasons; and recommending models and hyperparameters to handle special or missing values.
1. A computer-implemented method for managing missing values for a machine learning model, the method comprising:
identifying a missing value and a corresponding missing value reason for the missing value in a dataset with data point values, one or more of the data point values being associated with one or more features of the machine learning model;
applying an imputation method to generate an imputed feature for the missing value based on an imputation method;
replacing the missing value in the dataset with the imputed feature, the replacing comprising encoding the imputed feature with the corresponding missing value reason for the missing value for which the imputed feature is imputed; and
improving the machine learning model using the imputed feature.
2. The method of claim 1 , wherein data statistics are computed to determine that the missing value is missing.
3. The method of claim 2 , wherein in response to interaction with a domain expert, it is verified that the missing value is missing.
4. The method of claim 1 , wherein an encoding process is utilized to provide reasons for the missing value missing from the dataset.
5. The method of claim 4 , wherein the provided reasons and the imputed feature for the missing value and a corresponding reason for the missing value missing from the dataset are combined.
6. The method of claim 1 , wherein the imputation method comprises imputing a value of zero for the imputed feature.
7. The method of claim 1 , wherein the imputation method comprises at least one of a column mean value, a column media value, a column mode value, a column minimum value, or a column maximum value to impute the imputed feature for the missing value.
8. The method of claim 1 , wherein a deep autoencoder is used to impute the imputed feature for the missing value.
9. The method of claim 4 , wherein one or more missing or special value handling recommendation modules are utilized to recommend at least one of an imputation method for imputing the feature value of the first data point and an encoding process for providing the reasons for the first data point missing from the dataset.
10. The method of claim 9 , wherein an on-line or offline recommendation module is used to recommend missing or special value handling methods and hyperparameters, wherein the hyperparameters are used to by the imputation method which is selected by a user.
11. A system comprising:
at least one programmable processor; and
a non-transitory machine-readable medium storing instructions that, when executed by the at least one programmable processor, cause the at least one programmable processor to perform operations comprising:
identifying a missing value and a corresponding missing value reason for the missing value in a dataset with data point values, one or more of the data point values being associated with one or more features of the machine learning model;
applying an imputation method to generate an imputed feature for the missing value based on an imputation method;
replacing the missing value in the dataset with the imputed feature, the replacing comprising encoding the imputed feature with the corresponding missing value reason for the missing value for which the imputed feature is imputed; and
improving the machine learning model using the imputed feature.
12. The system of claim 11 , wherein data statistics are computed to determine that the missing value is missing.
13. The system of claim 12 , wherein in response to interaction with a domain expert, it is verified that the missing value is missing.
14. The system of claim 11 , wherein an encoding process is utilized to provide reasons for the missing value missing from the dataset.
15. The system of claim 14 , wherein the provided reasons and the imputed feature for the missing value and a corresponding reason for the missing value missing from the dataset are combined.
16. A computer program product comprising a non-transitory machine-readable medium storing instructions that, when executed by at least one programmable processor, cause the at least one programmable processor to perform operations comprising:
identifying a missing value and a corresponding missing value reason for the missing value in a dataset with data point values, one or more of the data point values being associated with one or more features of the machine learning model;
applying an imputation method to generate an imputed feature for the missing value based on an imputation method;
replacing the missing value in the dataset with the imputed feature, the replacing comprising encoding the imputed feature with the corresponding missing value reason for the missing value for which the imputed feature is imputed; and
improving the machine learning model using the imputed feature.
17. The computer program product of claim 16 , wherein data statistics are computed to determine that the missing value is missing.
18. The computer program product of claim 17 , wherein in response to interaction with a domain expert, it is verified that the missing value is missing.
19. The computer program product of claim 16 , wherein an encoding process is utilized to provide reasons for the missing value missing from the dataset.
20. The computer program product of claim 19 , wherein the provided reasons and the imputed feature for the missing value and a corresponding reason for the missing value missing from the dataset are combined.