Feature selection for deviation analysis
The present disclosure provides for accurate and efficient identification of candidate features for an input dataset comprising one or more continuous features and one or more categorical features is obtained. A number of categorical feature categories based on the one or more categorical features is determined. Record counts for each of the categorical feature categories are determined. Skew statistics for each category are determined based on the record counts for each of the categorical feature categories. Cardinality skew factors for each of the one or more categorical features are then determined based on the record counts and the skew statistics. A number of the one or more categorical features having the highest cardinality skew factors are selected from among the cardinality skew factors. Then, a top contributor deviation analysis is performed using the selected number of the categorical features having the highest cardinality skew factors.
1. A computer system, comprising:
one or more processors; and
one or more non-transitory machine-readable medium coupled to the one or more processors and storing computer program code comprising sets of instructions for executable by the one or more processors to:
obtain an input dataset comprising one or more continuous features and one or more categorical features;
determine a number of categorical feature categories based on the one or more categorical features;
determine record counts for each of the categorical feature categories;
calculate skew statistics for each category based on the record counts for each of the categorical feature categories;
determine cardinality skew factors for each of the one or more categorical features based on the record counts for each of the categorical feature categories and the skew statistics for each category, wherein the cardinality-skew factor indicates the distribution of a categorical features categories;
select a number of the one or more categorical features having the highest cardinality skew factors from among the cardinality skew factors for each of the one or more categorical features; and
distribute a top contributor deviation analysis across multiple cloud computing instances;
perform the top contributor deviation analysis on the multiple cloud computing instances using the selected number of the one or more categorical features having the highest cardinality skew factors.
2. The computer system of claim 1 , wherein the one or more categorical features have values that are within a plurality of categories corresponding to the one or more categorical features.
3. The computer system of claim 1 , wherein the computer program code further comprises sets of instructions executable by the one or more processors to:
normalize the skew statistic across all the categorical feature categories; and
normalize the record counts for each of the categorical feature categories across all the categorical feature categories.
4. The computer system of claim 1 , wherein the computer program code further comprises sets of instructions executable by the one or more processors to:
map continuous features of the one or more continuous features to the number of the one or more categorical features having the highest cardinality skew factors.
5. The computer system of claim 1 , wherein the computer program code further comprises sets of instructions executable by the one or more processors to:
order the one or more categorical features based on their corresponding cardinality skew factors.
6. The computer system of claim 1 , wherein the top contributor deviation analysis identifies top contributing features to a particular target feature.
7. The computer system of claim 5 , wherein performing the top contributor deviation analysis further comprises producing a deviation score for each of the one or more categorical features relative to the one or more continuous features of the input dataset and wherein ordering the one or more categorical features is further based on the deviation score.
8. One or more non-transitory computer-readable medium storing computer program code comprising sets of instructions to:
obtain an input dataset comprising one or more continuous features and one or more categorical features;
determine a number of categorical feature categories based on the one or more categorical features;
determine record counts for each of the categorical feature categories;
calculate skew statistics for each category based on the record counts for each of the categorical feature categories;
determine cardinality skew factors for each of the one or more categorical features based on the record counts for each of the categorical feature categories and the skew statistics for each category, wherein the cardinality-skew factor indicates the distribution of a categorical features categories;
select a number of the one or more categorical features having the highest cardinality skew factors from among the cardinality skew factors for each of the one or more categorical features; and
distribute a top contributor deviation analysis across multiple cloud computing instances;
perform the top contributor deviation analysis on the multiple cloud computing instances using the selected number of the one or more categorical features having the highest cardinality skew factors.
9. The non-transitory computer-readable medium of claim 8 , wherein the one or more categorical features have values that are within a plurality of categories corresponding to the one or more categorical features.
10. The non-transitory computer-readable medium of claim 8 , wherein the computer program code further comprises sets of instructions to:
normalize the skew statistic across all the categorical feature categories; and
normalize the record counts for each of the categorical feature categories across all the categorical feature categories.
11. The non-transitory computer-readable medium of claim 8 , wherein the computer program code further comprises sets of instructions to:
map continuous features of the one or more continuous features to the number of the one or more categorical features having the highest cardinality skew factors.
12. The non-transitory computer-readable medium of claim 8 , wherein the computer program code further comprises sets of instructions to:
order the one or more categorical features based on their corresponding cardinality skew factors.
13. The non-transitory computer-readable medium of claim 8 , wherein the top contributor deviation analysis identifies top contributing features to a particular target feature.
14. A computer-implemented method, comprising:
obtaining an input dataset comprising one or more continuous features and one or more categorical features;
determining a number of categorical feature categories based on the one or more categorical features;
determining record counts for each of the categorical feature categories;
calculating skew statistics for each category based on the record counts for each of the categorical feature categories;
determining cardinality skew factors for each of the one or more categorical features based on the record counts for each of the categorical feature categories and the skew statistics for each category, wherein the cardinality-skew factor indicates the distribution of a categorical features categories;
selecting a number of the one or more categorical features having the highest cardinality skew factors from among the cardinality skew factors for each of the one or more categorical features; and
distributing a top contributor deviation analysis across multiple cloud computing instances;
performing the top contributor deviation analysis on the multiple cloud computing instances using the selected number of the one or more categorical features having the highest cardinality skew factors.
15. The non-transitory computer-readable medium of claim 12 , wherein performing the top contributor deviation analysis further comprises producing a deviation score for each of the one or more categorical features relative to the one or more continuous features of the input dataset and wherein ordering the one or more categorical features is further based on the deviation score.
16. The computer-implemented method of claim 14 , wherein the one or more categorical features have values that are within a plurality of categories corresponding to the one or more categorical features.
17. The computer-implemented method of claim 14 , further comprising:
normalizing the skew statistic across all the categorical feature categories; and
normalizing the record counts for each of the categorical feature categories across all the categorical feature categories.
18. The computer-implemented method of claim 14 , further comprising:
mapping continuous features of the one or more continuous features to the number of the one or more categorical features having the highest cardinality skew factors.
19. The computer-implemented method of claim 14 , further comprising:
ordering the one or more categorical features based on their corresponding cardinality skew factors.
20. The computer-implemented method of claim 19 , wherein performing the top contributor deviation analysis further comprises producing a deviation score for each of the one or more categorical features relative to the one or more continuous features of the input dataset and wherein ordering the one or more categorical features is further based on the deviation score.