Feature selection for model training
Systems and methods include determination of a first plurality of sets of data, each including values associated with respective ones of a first plurality of features, partial training of a first machine-learning model based on the first plurality of sets of data, determination of one or more of the first plurality of features to remove based on the partially-trained first machine-learning model, removal of the one or more of the first plurality of features to generate a second plurality of sets of data, partial training of a second machine-learning model based on the second plurality of sets of data, determination that a performance of the partially-trained second machine-learning model is less than a threshold, addition, in response to the determination, of the one or more of the first plurality of features to the second plurality of sets of data, and training of the partially-trained first machine-learning model based on the first plurality of sets of data.
1. A system comprising:
a memory storing processor-executable program code; and
a processing unit to execute the processor-executable program code to cause the system to:
determine a data structure representing a plurality of rows, the data structure associating each nonzero value of each of the plurality of rows with an index of one of a first plurality of features represented by the nonzero value, and including a row index indicating locations of nonzero values of each of the plurality of rows within the data structure;
partially train a first machine-learning model based on the data structure;
determine one of the first plurality of features to remove from the data structure based on the partially-trained first machine-learning model;
in response to the determination of the one of the first plurality of features to remove from the data structure, modify the data structure by removal of each nonzero value associated with an index of the one of the first plurality of features from the data structure and modify the row index to indicate new locations of nonzero values of each of the plurality of rows within the data structure;
partially train a second machine-learning model based on the modified data structure;
determine that a performance of the partially-trained second machine-learning model is less than a threshold; and
in response to the determination, train the partially-trained first machine-learning model based on the data structure.
2. The system according to claim 1 , wherein partial training of the first machine-learning model comprising executing a predefined number of training iterations, and
wherein training of the partially-trained first machine-learning model comprises determining that the trained first machine-learning model meets a stopping criterion.
3. The system according to claim 2 , wherein determination of the one of the first plurality of features to remove from the data structure based on the partially-trained first machine-learning model comprises:
determination of importance values associated with each of the first plurality of features based on a subset of the data structure and on the partially-trained first machine-learning model.
4. The system according to claim 3 , wherein the data structure comprises a first array and a second array, wherein each row of the first array includes a nonzero value of one of the plurality of rows and an index of one of the first plurality of features represented by the nonzero value,
wherein each row of the second array includes a row index representing a starting location in the first array of the nonzero values of a corresponding one of the plurality of rows, and
wherein modification of the data structure comprises:
removal of each row of the first array including the index of the one of the first plurality of features;
after removal of each row of the first array including the index of the one of the first plurality of features, shifting of the rows of the first array to fill the removed rows; and
modification of the row index to indicate new starting locations in the first array of the nonzero values of corresponding ones of the plurality of rows.
5. The system according to claim 2 , wherein the first machine-learning model and the second machine-learning model comprise decision tree models, and wherein the training iterations comprise boosting iterations.
6. The system according to claim 1 , wherein determination of the one of the first plurality of features to remove from the data structure based on the partially-trained first machine-learning model comprises:
determination of importance values associated with each of the first plurality of features based on a subset of the data structure and on the partially-trained first machine-learning model.
7. The system according to claim 1 , wherein the data structure comprises a first array and a second array, wherein each row of the first array includes a nonzero value of one of the plurality of rows and an index of one of the first plurality of features represented by the nonzero value,
wherein each row of the second array includes a row index representing a starting location in the first array of the nonzero values of a corresponding one of the plurality of rows, and
wherein modification of the data structure comprises:
removal of each row of the first array including the index of the one of the first plurality of features;
after removal of each row of the first array including the index of the one of the first plurality of features, shifting of the rows of the first array to fill the removed rows; and
modification of the row index to indicate new starting locations in the first array of the nonzero values of corresponding ones of the plurality of rows.
8. A method comprising:
determining a data structure representing a plurality of rows, the data structure associating each nonzero value of each of the plurality of rows with an index of one of a first plurality of features represented by the nonzero value, and including a row index indicating locations of nonzero values of each of the plurality of rows within the data structure;
partially training a first machine-learning model based on the data structure;
determining one of the first plurality of features to remove from the data structure based on the partially-trained first machine-learning model;
in response to the determination of the one of the first plurality of features to remove from the data structure, modify the data structure by removal of each nonzero value associated with an index of the one of the first plurality of features from the data structure and modify the row index to indicate new locations of nonzero values of each of the plurality of rows within the data structure;
partially training a second machine-learning model based on the modified data structure;
determining that a performance of the partially-trained second machine-learning model is less than a threshold; and
in response to the determination, train the partially-trained first machine-learning model based on the data structure.
9. The method according to claim 8 , wherein partial training of the first machine-learning model comprising executing a predefined number of training iterations, and
wherein training of the partially-trained first machine-learning model comprises determining that the trained first machine-learning model meets a stopping criterion.
10. The method according to claim 9 , wherein determining the one of the first plurality of features to remove from the data structure based on the partially-trained first machine-learning model comprises:
determining importance values associated with each of the first plurality of features based on a subset of the data structure and on the partially-trained first machine-learning model.
11. The method according to claim 10 , wherein the data structure comprises a first array and a second array, wherein each row of the first array includes a nonzero value of one of the plurality of rows and an index of one of the first plurality of features represented by the nonzero value,
wherein each row of the second array includes a row index representing a starting location in the first array of the nonzero values of a corresponding one of the plurality of rows, and
wherein modifying the data structure comprises:
removing each row of the first array including the index of the one of the first plurality of features;
after removing each row of the first array including the index of the one of the first plurality of features, shifting of rows of the first array to fill the removed rows; and
modifying the row index to indicate new starting locations in the first array of the nonzero values of corresponding ones of the plurality of rows.
12. The method according to claim 9 , wherein the first machine-learning model and the second machine-learning model comprise decision tree models, and wherein the training iterations comprise boosting iterations.
13. The method according to claim 8 , wherein determining the one of the first plurality of features to remove from the data structure based on the partially-trained first machine-learning model comprises:
determining importance values associated with each of the first plurality of features based on a subset of the data structure and on the partially-trained first machine-learning model.
14. The method according to claim 8 , wherein the data structure comprises a first array and a second array, wherein each row of the first array includes a nonzero value of one of the plurality of rows and an index of one of the first plurality of features represented by the nonzero value,
wherein each row of the second array includes a row index representing a starting location in the first array of the nonzero values of a corresponding one of the plurality of rows, and
wherein modifying the data structure comprises:
removing each row of the first array including the index of the one of the first plurality of features;
after removing each row of the first array including the index of the one of the first plurality of features, shifting the rows of the first array to fill the removed rows; and
modifying the row index to indicate new starting locations in the first array of the nonzero values of corresponding ones of the plurality of rows.
15. A non-transitory medium storing processor-executable program code executable by a processing unit of a computing system to cause the computing system to:
determine a data structure representing a plurality of rows, the data structure associating each nonzero value of each of the plurality of rows with an index of one of a first plurality of features represented by the nonzero value, and including a row index indicating locations of nonzero values of each of the plurality of rows within the data structure;
partially train a first machine-learning model based on the data structure;
determine one of the first plurality of features to remove from the data structure based on the partially-trained first machine-learning model;
in response to the determination of the one of the first plurality of features to remove from the data structure, modify the data structure by removal of each nonzero value associated with an index of the one of the first plurality of features from the data structure and modify the row index to indicate new locations of nonzero values of each of the plurality of rows within the data structure;
partially train a second machine-learning model based on the modified data structure;
determine that a performance of the partially-trained second machine-learning model is less than a threshold; and
in response to the determination, train the partially-trained first machine-learning model based on the data structure.
16. The non-transitory medium according to claim 15 , wherein partial training of the first machine-learning model comprising executing a predefined number of training iterations, and
wherein training of the partially-trained first machine-learning model comprises determining that the trained first machine-learning model meets a stopping criterion.
17. The non-transitory medium according to claim 16 , wherein determination of the one of the first plurality of features to remove from the data structure based on the partially-trained first machine-learning model comprises:
determination of importance values associated with each of the first plurality of features based on a subset of the data structure and on the partially-trained first machine-learning model.
18. The non-transitory medium according to claim 17 , wherein the data structure comprises a first array and a second array, wherein each row of the first array includes a nonzero value of one of the plurality of rows and an index of one of the first plurality of features represented by the nonzero value,
wherein each row of the second array includes a row index representing a starting location in the first array of the nonzero values of a corresponding one of the plurality of rows, and
wherein modification of the data structure comprises:
removal of each row of the first array including the index of the one of the first plurality of features;
after removal of each row of the first array including the index of the one of the first plurality of features, shifting of the rows of the first array to fill the removed rows; and
modification of the row index to indicate new starting locations in the first array of the nonzero values of corresponding ones of the plurality of rows.
19. The non-transitory medium according to claim 16 , wherein the first machine-learning model and the second machine-learning model comprise decision tree models, and wherein the training iterations comprise boosting iterations.
20. The non-transitory medium according to claim 15 , wherein the data structure comprises a first array and a second array, wherein each row of the first array includes a nonzero value of one of the plurality of rows and an index of one of the first plurality of features represented by the nonzero value,
wherein each row of the second array includes a row index representing a starting location in the first array of the nonzero values of a corresponding one of the plurality of rows, and
wherein modification of the data structure comprises:
removal of each row of the first array including the index of the one of the first plurality of features;
after removal of each row of the first array including the index of the one of the first plurality of features, shifting of the rows of the first array to fill the removed rows; and
modification of the row index to indicate new starting locations in the first array of the nonzero values of corresponding ones of the plurality of rows.