Method and system of performing data imbalance detection and correction in training a machine-learning model

1. A data processing system comprising:

a processor; and

a memory in communication with the processor, the memory comprising executable instructions that, when executed by the processor cause the data processing system to perform functions of:

receiving a request to train a machine-learning (ML) model;

receiving access to a dataset for use in training the ML model;

identifying a feature of the dataset for which data imbalance detection is to be performed;

examining the dataset to determine a distribution of the feature across the dataset;

determining if the distribution of the feature across the dataset indicates a data imbalance;

upon determining that the distribution of the feature across the dataset indicates the data imbalance, identifying a desired distribution for the identified feature;

selecting a subset of the dataset that corresponds with the selected feature and the desired distribution; and

using the subset to train the ML model.

2. The data processing system of claim 1 , wherein the functions are performed without user input.

3. The data processing system of claim 1 , wherein the functions are performed with some user input.

4. The data processing system of claim 1 , wherein examining the dataset to determine a distribution of the feature includes performing a statistical analysis on the dataset to determine the distribution of the feature across one or more categories available for the feature.

5. The data processing system of claim 1 , wherein the dataset includes at least one of an input training dataset, a training subset of the input training dataset, a validation subset of the input training dataset, and an outcome dataset.

6. The data processing system of claim 1 , wherein the feature includes a label feature of the dataset.

7. The data processing system of claim 1 , wherein the executable instructions when executed by the processor further cause the data processing system to perform functions of:

examining the subset to determine if a subset data imbalance exists;

upon determining the subset data imbalance, performing a data imbalance correction on the subset to create a corrected subset; and

repeating a process of examining the corrected subset and performing data imbalance on the corrected subset until a desired subset is created.

8. A method for performing bias detection and correction in training a ML model, the method comprising:

receiving a request to train the ML model;

receiving access to a dataset for use in training the ML model;

identifying a feature of the dataset for which data imbalance detection is to be performed;

examining the dataset to determine a distribution of the feature across the dataset;

determining if the distribution of the feature across the dataset indicates a data imbalance;

upon determining that the distribution of the feature across the dataset indicates the data imbalance, identifying a desired distribution for the identified feature;

selecting a subset of the dataset that corresponds with the selected feature and the desired distribution; and

using the subset to train the ML model.

9. The method of claim 8 , wherein steps of the method are performed without user input.

10. The method of claim 8 , wherein steps of the method are performed with some user input.

11. The method of claim 8 , wherein examining the dataset to determine a distribution of the feature includes performing a statistical analysis on the dataset to determine the distribution of the feature across one or more categories available for the feature.

12. The method of claim 8 , wherein the dataset includes at least one of an input training dataset, a training subset of the input training dataset, a validation subset of the input training dataset, and an outcome dataset.

13. The method of claim 8 , wherein the feature includes a label feature of the dataset.

14. The method of claim 8 , further comprising:

examining the subset to determine if a subset data imbalance exists;

upon determining the subset data imbalance, performing a data imbalance correction on the subset to create a corrected subset; and

repeating a process of examining the corrected subset and performing data imbalance on the corrected subset until a desired subset is created.

15. A non-transitory computer readable medium on which are stored instructions that, when executed cause a programmable device to:

receive a request to train a machine-learning (ML) model;

receive access to a dataset for use in training the ML model;

identify a feature of the dataset for which data imbalance detection is to be performed;

examine the dataset to determine a distribution of the feature across the dataset;

determine if the distribution of the feature across the dataset indicates a data imbalance;

upon determining that the distribution of the feature across the dataset indicates the data imbalance, identify a desired distribution for the identified feature;

select a subset of the dataset that corresponds with the selected feature and the desired distribution; and

use the subset to train the ML model.

16. The non-transitory computer readable medium of claim 15 , wherein the instructions when executed cause the programmable device to perform steps without user input.

17. The non-transitory computer readable medium of claim 15 , wherein the instructions when executed cause the programmable device to perform steps with some user input.

18. The non-transitory computer readable medium of claim 15 , wherein examining the dataset to determine a distribution of the feature includes performing a statistical analysis on the dataset to determine the distribution of the feature across one or more categories available for the feature.

19. The non-transitory computer readable medium of claim 15 , wherein the feature includes a label feature of the dataset.

20. The non-transitory computer readable medium of claim 15 , wherein the instructions, when executed cause the programmable device to:

examine the subset to determine if the data imbalance exists;

upon determining the subset data imbalance, perform a data imbalance correction on the subset to create a corrected subset; and

repeat a process of examining the corrected subset and performing data imbalance on the corrected subset until a desired subset is created.