Methods and systems for feature selection in machine learning based on feature contribution and model fitness
View Patent ↗Methods and systems are provided for feature selection in machine learning, in which the features selected for inclusion in a prediction rule are selected based on statistical metric(s) of feature contribution and/or model fitness.
1. A method for selecting features for a final prediction rule predictive of an outcome with respect to a medical condition, said method comprising:
performing with a computer-implemented machine learning tool:
(a) generating a prediction rule based on training data for a cohort of patients whose outcomes with respect to said medical condition are at least partially known, wherein for each patient the data comprises measurements for a set of features and the outcome with respect to said medical condition for said patient to the extent known, wherein in a first iteration of (a) said set of features includes n features with n greater than or equal to 3 with n being decremented by one in each subsequent iteration of (a);
(b) determining a fitness value for said prediction rule, wherein said determining a fitness value comprises summing a concordance index (CI) of said prediction rule with a product of a sensitivity and a specificity of said prediction rule;
(c) determining a value of contribution to said prediction rule for each of said features in said set of features;
(d) removing a feature from consideration from said set of features based on the values of contribution, wherein the feature having the lowest value of contribution is removed;
(e) iterating (a)-(d) in order to produce n prediction rules and n fitness values; and
(f) selecting, based on the fitness values for said n prediction rules, one of said n prediction rules as said final prediction rule predictive of the outcome with respect to said medical condition, wherein of said n prediction rules said final prediction rule has the highest predictive ability with respect to the outcome with respect to said medical condition as indicated by said fitness values; and
evaluating data for a patient with a computer implementation of said final prediction rule to produce a value predictive of the patient's outcome with respect to said medical condition.
2. The method of claim 1 , wherein stages (a)-(f) are performed in that order.
3. The method of claim 1 , wherein said determining a value of contribution comprises determining a contribution value for each feature i by multiplying a weight of said feature i in said prediction rule with a measure of the discriminative ability of said feature i as observed in the data used to generate said prediction rule.
4. The method of claim 1 , wherein said determining a value of contribution comprises determining a contribution value for each feature i by multiplying a weight of said feature i in said prediction rule with a concordance index (CI) for said feature i.
5. The method of claim 1 , wherein said iterating (a)-(d) comprises:
repeating (a) and (b) as long as said set of features includes 1 or more features; and
repeating (c) and (d) as long as said set of features includes 2 or more features.
6. The method of claim 1 , wherein said generating a prediction rule comprises generating a prediction rule based on support vector regression for censored data with a linear kernel.
7. The method of claim 1 , wherein said determining a value of contribution comprises determining a contribution value for each feature by multiplying a weight of said feature in said prediction rule with a variance or standard deviation of said measurements of said feature as observed in the training data used to generate said prediction rule.
8. A method for selecting features for a final prediction rule predictive of an outcome with respect to a medical condition, said method comprising:
performing with a computer-implemented machine learning tool:
(a) generating a prediction rule based on training data for a cohort of patients whose outcomes with respect to said medical condition are at least partially known, wherein for each patient the data comprises measurements for a set of features and the outcome with respect to said medical condition for said patient to the extent known, wherein in a first iteration of (a) said set of features includes n features with n greater than or equal to 3 with n being decremented by one in each subsequent iteration of (a);
(b) determining a fitness value for said prediction rule, wherein said determining a fitness value comprises calculating the following risk functional:
R ( k, L )=fitness/(1−sqrt( ( k (log( L/k )+log( L )/2)/ L ))
where L is a number of instances in said data for said set of n features, k is a VC-dimension of a set of functions from which said prediction rule is selected, and fitness is a function that evaluates quality of fit of said prediction rule;
(c) determining a value of contribution to said prediction rule for each of said features in said set of features;
(d) removing a feature from consideration from said set of features based on the values of contribution, wherein the feature having the lowest value of contribution is removed;
(e) iterating (a)-(d) in order to produce n prediction rules and n fitness values; and
(f) selecting, based on the fitness values for said n prediction rules, one of said n prediction rules as said final prediction rule predictive of the outcome with respect to said medical condition, wherein of said n prediction rules said final prediction rule has the highest predictive ability with respect to the outcome with respect to said medical condition as indicated by said fitness values; and
evaluating data for a patient with a computer implementation of said final prediction rule to produce a value predictive of the patient's outcome with respect to said medical condition.
9. The method of claim 8 , wherein said generating a prediction rule comprises generating a prediction rule based on support vector regression for censored data with a linear kernel.
10. The method of claim 8 , wherein said determining a value of contribution comprises determining a contribution value for each feature by multiplying a weight of said feature in said prediction rule with a variance or standard deviation of said measurements of said feature as observed in the training data used to generate said prediction rule.
11. An apparatus for determining a risk of occurrence of an outcome with respect to a medical condition in a patient, said apparatus comprising:
a computer implementation of a final prediction rule predictive of said medical condition, wherein said final prediction rule is based on features selected through machine learning, said machine learning comprising performing with a computer-implemented machine learning tool (a) generating a prediction rule based on training data for a cohort of patients whose outcomes with respect to said medical condition are at least partially known, wherein for each patient the data comprises measurements for a set of features and the outcome with respect to said medical condition for said patient to the extent known, wherein in a first iteration of (a) said set includes n features with n greater than or equal to 3 with n being decremented by one in each subsequent iteration of (a), (b) determining a fitness value for said prediction rule, wherein said determining a fitness value comprises summing a concordance index (CI) of said prediction rule with a product of a sensitivity and a specificity of said prediction rule, (c) determining a value of contribution to said prediction rule for each of said features in said set of features, (d) removing a feature from consideration from said set of features based on the values of contribution, wherein the feature having the lowest value of contribution is removed, (e) iterating (a)-(d) in order to produce n prediction rules and n fitness values, and (f) selecting, based on the fitness values for said n prediction rules, one of said n prediction rules as said final prediction rule, wherein of said n prediction rules said final prediction rule has the highest predictive ability with respect to the outcome with respect to said medical condition as indicated by said fitness values, wherein said computer implementation of said final prediction rule:
receives data for said patient; and
evaluates said data for said patient according to said final prediction rule, thereby determining a value predictive of the patient's outcome with respect to said medical condition.
12. The apparatus of claim 11 , wherein said machine learning determines a value of contribution for each feature i by multiplying a weight of said feature i in said prediction rule with a measure of the discriminative ability of said feature i as observed in the data used to generate said prediction rule.
13. The apparatus of claim 11 , wherein said machine learning determines a value of contribution for each feature i by multiplying a weight of said feature i in said prediction rule with a concordance index (CI) for said feature i.
14. The apparatus of claim 11 , wherein said generating a prediction rule comprises generating a prediction rule based on support vector regression for censored data with a linear kernel.
15. The apparatus of claim 11 , wherein said determining a value of contribution comprises determining a contribution value for each feature by multiplying a weight of said feature in said prediction rule with a variance or standard deviation of said measurements of said feature as observed in the training data used to generate said prediction rule.
16. An apparatus for determining a risk of occurrence of an outcome with respect to a medical condition in a patient, said apparatus comprising:
a computer implementation of a final prediction rule predictive of said medical condition, wherein said final prediction rule is based on features selected through machine learning, said machine learning comprising performing with a computer-implemented machine learning tool (a) generating a prediction rule based on training data for a cohort of patients whose outcomes with respect to said medical condition are at least partially known, wherein for each patient the data comprises measurements for a set of features and the outcome with respect to said medical condition for said patient to the extent known, wherein in a first iteration of (a) said set includes n features with n greater than or equal to 3 with n being decremented by one in each subsequent iteration of (a), (b) determining a fitness value for said prediction rule, wherein said determining a fitness value comprises calculating the following risk functional:
R ( k, L )=fitness/(1−sqrt( ( k (log( L/k )+log( L )+log( L )/2)/ L ))
where L is a number of instances in said data for said set of n features, k is a VC-dimension of a set of functions from which said prediction rule is selected, and fitness is a function that evaluates quality of fit of said prediction rule, (c) determining a value of contribution to said prediction rule for each of said features in said set of features, (d) removing a feature from consideration from said set of features based on the values of contribution, wherein the feature having the lowest value of contribution is removed, (e) iterating (a)-(d) in order to produce n prediction rules and n fitness values, and (f) selecting, based on the fitness values for said n-prediction rules, one of said n prediction rules as said final prediction rule, wherein of said n prediction rules said final prediction rule has the highest predictive ability with respect to the outcome with respect to said medical condition as indicated by said fitness values, wherein said computer implementation of said final prediction rule:
receives data for said patient; and
evaluates said data for said patient according to said final prediction rule, thereby determining a value predictive of the patient's outcome with respect to said medical condition.
17. The apparatus of claim 16 , wherein said generating a prediction rule comprises generating a prediction rule based on support vector regression for censored data with a linear kernel.
18. The apparatus of claim 16 , wherein said determining a value of contribution comprises determining a contribution value for each feature by multiplying a weight of said feature in said prediction rule with a variance or standard deviation of said measurements of said feature as observed in the training data used to generate said prediction rule.
19. A computer readable medium comprising computer executable instructions recorded thereon for performing a method for selecting features for a final prediction rule predictive of an outcome with respect to a medical condition, the method comprising:
performing with a computer-implemented machine learning tool:
(a) generating a prediction rule based on training data for a cohort of patients whose outcomes with respect to said medical condition are at least partially known, wherein for each patient the data comprises measurements for a set of features and the outcome with respect to said medical condition for said patient to the extent known, wherein in a first iteration of (a) said set includes n features with n greater than or equal to 3 with n being decremented by one in each subsequent iteration of (a);
(b) determining a fitness value for said prediction rule, wherein said determining a fitness value comprises summing a concordance index (CI) of said prediction rule with a product of a sensitivity and a specificity of said prediction rule;
(c) determining a value of contribution to said prediction rule for each of said features in said set of features;
(d) removing a feature from consideration from said set of features based on the values of contribution, wherein the feature having the lowest value of contribution is removed;
(e) iterating (a)-(d) to produce n prediction rules and n fitness values;
(f) selecting, based on the fitness values for said n prediction rules, one of said n prediction rules as the basis for said final prediction rule predictive of the outcome with respect to said medical condition, wherein of said n prediction rules said final prediction rule has the highest predictive ability with respect to the outcome with respect to said medical condition as indicated by said fitness values; and
evaluating data for a patient with a computer implementation of said final prediction rule to produce a value predictive of the patient's outcome with respect to said medical condition.
20. The computer readable medium of claim 19 , further comprising computer executable instructions recorded thereon for determining a value of contribution by performing the method comprising determining a contribution value for each feature i by multiplying a weight of said feature i in said prediction rule with a measure of the discriminative ability of said feature i as observed in the data used to generate said prediction rule.
21. The computer readable medium of claim 19 , further comprising computer executable instructions recorded thereon for determining a value of contribution by performing the method comprising determining a contribution value for each feature i by multiplying a weight of said feature i in said prediction rule with a concordance index (CI) for said feature i.
22. The computer readable medium of claim 19 , wherein said generating a prediction rule comprises generating a prediction rule based on support vector regression for censored data with a linear kernel.
23. The computer readable medium of claim 19 , wherein said determining a value of contribution comprises determining a contribution value for each feature by multiplying a weight of said feature in said prediction rule with a variance or standard deviation of said measurements of said feature as observed in the training data used to generate said prediction rule.
24. A computer readable medium comprising computer executable instructions recorded thereon for performing a method for selecting features for a final prediction rule predictive of an outcome with respect to a medical condition, the method comprising:
performing with a computer-implemented machine learning tool:
(a) generating a prediction rule based on training data for a cohort of patients whose outcomes with respect to said medical condition are at least partially known, wherein for each patient the data comprises measurements for a set of features and the outcome with respect to said medical condition for said patient to the extent known, wherein in a first iteration of (a) said set includes n features with n greater than or equal to 3 with n being decremented by one in each subsequent iteration of (a);
(b) determining a fitness value for said prediction rule, wherein said determining a fitness value comprises calculating the following risk functional:
R ( k, L )=fitness/(1−sqrt( ( k (log( L/k )+log( L )/2)/ L ))
where L is a number of instances in said data for said set of n features, k is a VC-dimension of a set of functions from which said prediction rule is selected, and fitness is a function that evaluates quality of fit of said prediction rule;
(c) determining a value of contribution to said prediction rule for each of said features in said set of features;
(d) removing a feature from consideration from said set of features based on the values of contribution, wherein the feature having the lowest value of contribution is removed;
(e) iterating (a)-(d) to produce n prediction rules and n fitness values;
(f) selecting, based on the fitness values for said n prediction rules, one of said n prediction rules as the basis for said final prediction rule predictive of the outcome with respect to said medical condition, wherein of said n prediction rules said final prediction rule has the highest predictive ability with respect to the outcome with respect to said medical condition as indicated by said fitness values; and
evaluating data for a patient with a computer implementation of said final prediction rule to produce a value predictive of the patient's outcome with respect to said medical condition.
25. The computer readable medium of claim 24 , wherein said generating a prediction rule comprises generating a prediction rule based on support vector regression for censored data with a linear kernel.
26. The computer readable medium of claim 24 , wherein said determining a value of contribution comprises determining a contribution value for each feature by multiplying a weight of said feature in said prediction rule with a variance or standard deviation of said measurements of said feature as observed in the training data used to generate said prediction rule.