Methods and systems for feature selection in machine learning based on feature contribution and model fitness

1 . A method for selecting features for a final prediction rule, said method comprising:

(a) generating a prediction rule based on data for a set of features, wherein initially said set of features includes n features;

(b) determining a fitness value for said prediction rule;

(c) determining a value of contribution to said prediction rule for each of said features in said set of features;

(d) removing a feature from consideration from said set of features based on the values of contribution;

(e) iterating (a)-(d) in order to produce n prediction rules and n fitness values; and

(f) selecting, based on the fitness values for said n prediction rules, one of said n models as said final prediction rule.

2 . The method of claim 1 , wherein stages (a)-(f) are performed in that order.

3 . The method of claim 1 , wherein determining a fitness value for said prediction rule comprises summing a concordance index (CI) of said prediction rule with a product of a sensitivity and a specificity of said prediction rule.

4 . The method of claim 1 , wherein determining a fitness value for said prediction rule comprises measuring a separation between one or more outcome value(s) predicted by said prediction rule and actual outcome(s).

5 . The method of claim 1 , wherein determining a fitness value for said prediction rule comprises calculating the following risk functional:

R ( k,l )=fitness/(1−sqrt(( k (log( l/k )+1)+log( l )/2)/ l ))

where l is a number of instances in said data for said set of n features, k is a VC-dimension of a set of functions from which said prediction rule is selected, and fitness is a function that evaluates quality of fit of said prediction rule.

6 . The method of claim 1 , wherein said determining a value of contribution comprises determining a contribution value for each feature i by multiplying a weight of said feature i in said prediction rule with a measure of the discriminative ability of said feature i as observed in the data used to generate said prediction rule.

7 . The method of claim 1 , wherein said determining a value of contribution comprises determining a contribution value for each feature i by multiplying a weight of said feature i in said prediction rule with a concordance index (CI) for said feature i.

8 . The method of claim 1 , wherein said iterating (a)-(d) comprises:

repeating (a) and (b) as long as said set of features includes 1 or more features; and

repeating (c) and (d) as long as said set of features includes 2 or more features.

9 . An apparatus for determining an outcome for an instance, said apparatus comprising:

a computer implementation of a final prediction rule, wherein said final prediction rule is based on features selected through machine learning, said machine learning comprising (a) generating a prediction rule based on data for a set of features, wherein initially said set includes n features, (b) determining a fitness value for said prediction rule, (c) determining a value of contribution to said prediction rule for each of said features in said set of features, (d) removing a feature from consideration from said set of features based on the values of contribution, (e) iterating (a)-(d) in order to produce n prediction rules and n fitness values, and (f) selecting, based on the fitness values for said n models, one of said n prediction rules as said final prediction rule, wherein said computer implementation of said final prediction rule is configured to:

receive data for an instance; and

evaluate said data for said instance according to said final prediction rule, thereby determining an outcome for said instance.

10 . The apparatus of claim 9 , wherein said machine learning determines a fitness value for said prediction rule by summing a concordance index (CI) of said prediction rule with a product of a sensitivity and a specificity of said prediction rule.

11 . The apparatus of claim 9 , wherein said machine learning determines a fitness value for said prediction rule by measuring a separation between one or more outcome value(s) predicted by said prediction rule and actual outcome(s).

12 . The apparatus of claim 9 , wherein said machine learning determines a fitness value for said prediction rule by calculating the following risk functional:

R ( k,l )=fitness/(1−sqrt(( k (log( l/k )+1)+log( l )/2)/ l ))

where l is a number of instances in said data for said set of n features, k is a VC-dimension of a set of functions from which said prediction rule is selected, and fitness is a function that evaluates quality of fit of said prediction rule.

13 . The apparatus of claim 9 , wherein said machine learning determines a value of contribution for each feature i by multiplying a weight of said feature i in said prediction rule with a measure of the discriminative ability of said feature i as observed in the data used to generate said prediction rule.

14 . The apparatus of claim 9 , wherein said machine learning determines a value of contribution for each feature i by multiplying a weight of said feature i in said prediction rule with a concordance index (CI) for said feature i.

15 . A computer readable medium comprising computer executable instructions recorded thereon for performing the method comprising:

(a) generating a prediction rule based on data for a set of features, wherein initially said set includes n features;

(b) determining a fitness value for said prediction rule;

(c) determining a value of contribution to said prediction rule for each of said features in said set of features;

(d) removing a feature from consideration from said set of features based on the values of contribution;

(e) iterating (a)-(d) to produce n prediction rules and n fitness values; and

(f) selecting, based on the fitness values for said n models, one of said n prediction rules as the basis for said final prediction rule.

16 . The computer readable medium of claim 15 , further comprising computer executable instructions recorded thereon for determining a fitness value for said prediction rule by performing the method comprising summing a concordance index (CI) of said prediction rule with a product of a sensitivity and a specificity of said prediction rule.

17 . The computer readable medium of claim 15 , further comprising computer executable instructions recorded thereon for determining a fitness value for said prediction rule by performing the method comprising measuring a separation between one or more outcome value(s) predicted by said prediction rule and actual outcome(s).

18 . The computer readable medium of claim 15 , further comprising computer executable instructions recorded thereon for determining a fitness value for said prediction rule by performing the method comprising calculating the following risk functional:

R ( k,l )=fitness/(1−sqrt(( k (log( l/k )+1)+log( l )/2)/ l ))

where l is a number of instances in said data for said set of n features, k is a VC-dimension of a set of functions from which said prediction rule is selected, and fitness is a function that evaluates quality of fit of said prediction rule.

19 . The computer readable medium of claim 15 , further comprising computer executable instructions recorded thereon for determining a value of contribution by performing the method comprising determining a contribution value for each feature i by multiplying a weight of said feature i in said prediction rule with a measure of the discriminative ability of said feature i as observed in the data used to generate said prediction rule.

20 . The computer readable medium of claim 15 , further comprising computer executable instructions recorded thereon for determining a value of contribution by performing the method comprising determining a contribution value for each feature i by multiplying a weight of said feature i in said prediction rule with a concordance index (CI) for said feature i.