Multi-stage training of machine learning models

1. A system comprising:

storage media;

one or more processors; and

one or more programs stored in the storage media and configured for execution by the one or more processors, the one or more programs comprising instructions for:

using one or more first machine learning techniques to generate a first prediction model based on first training data, wherein the first prediction model enables determining a likelihood of whether exploits will be developed for software vulnerabilities;

wherein the first training data comprises a first plurality of training instances, and wherein each training instance of the first plurality of training instances comprises one or more features corresponding to a respective software vulnerability;

for each training instance of the first plurality of training instances:

using the first prediction model to generate a corresponding score, and

adding said each training instance to second training data only if the corresponding score is greater than a first threshold value;

wherein the second training data is a strict subset of the first training data; and

using one or more second machine learning techniques to generate a second prediction model based on the second training data, wherein the second prediction model enables determining a likelihood of whether exploits will be developed for software vulnerabilities.

2. The system of claim 1 , wherein for each training instance, the corresponding score indicates a likelihood that an exploit will be developed for a software vulnerability corresponding to said each training instance.

3. The system of claim 2 , wherein scores below the first threshold value indicate that it is unlikely that an exploit will be developed for a software vulnerability.

4. The system of claim 1 , wherein the one or more programs further comprise instructions for:

causing application of the first prediction model to first input data that corresponds to a first plurality of software vulnerabilities, wherein each software vulnerability of the first plurality of software vulnerabilities has yet to have an exploit developed for said each software vulnerability;

receiving, based on the application of the first prediction model to the first input data, first output data that indicates, for each software vulnerability of the first plurality of software vulnerabilities, a prediction of whether an exploit will be developed for said each software vulnerability;

for each prediction indicated in the first output data, adding, to second input data, a software vulnerability corresponding to said each prediction only if said each prediction is above a second threshold value;

causing application of the second prediction model to the second input data; and

receiving, based on the application of the second prediction model to the second input data, second output data that indicates, for each software vulnerability in the second plurality of software vulnerabilities, a prediction of whether an exploit will be developed for said each software vulnerability.

5. The system of claim 4 , wherein the first threshold value is different from the second threshold value.

6. The system of claim 4 , wherein the one or more programs further comprise instructions for:

causing generation of a third prediction model based on third training data, wherein the third training data is a subset of the first training data, and wherein each training instance of the third training data is predicted to have an exploit developed, wherein the third prediction model enables determining a likelihood of whether exploits that have yet to be developed will be used in an attack;

wherein the third training data comprises a second plurality of training instances, and wherein each training instance of the second plurality of training instances comprises one or more features corresponding to a respective software vulnerability;

for each training instance of the second plurality of training instances:

using the third prediction model to generate a second score, and adding said each training instance to fourth training data only if the second score is greater than a third threshold value; and

causing generation of a fourth prediction model based on the fourth training data, wherein the fourth prediction model enables determining a likelihood of whether exploits that have yet to be developed will be used in an attack.

7. The system of claim 6 , wherein the one or more programs further comprise instructions for:

causing application of the third prediction model to third input data that corresponds to a third plurality of software vulnerabilities, wherein the third plurality of software vulnerabilities is a subset of the second plurality of software vulnerabilities, and wherein each software vulnerability of the third plurality of software vulnerabilities is predicted to have an exploit developed for said each software vulnerability of the third plurality of software vulnerabilities;

receiving, based on the application of the third prediction model to the third input data, third output data that indicates, for each software vulnerability of the third plurality of software vulnerabilities, a first prediction of whether a corresponding exploit that is predicted to be developed for said each software vulnerability is also predicted to be used in an attack;

causing application of the fourth prediction model to fourth input data that corresponds to a fourth plurality of software vulnerabilities, wherein the fourth plurality of software vulnerabilities is determined based on removing, from the third input data, data that corresponds to a subset of the third plurality of software vulnerabilities, wherein each software vulnerability in the subset of the third plurality of software vulnerabilities is predicted to have an exploit developed for said each software vulnerability in the subset, and wherein the corresponding exploit that is predicted to be developed for said each software vulnerability in the subset is predicted not to be used in an attack; and

receiving, based on the application of the fourth prediction model to the fourth input data, fourth output data that indicates, for each software vulnerability in the fourth plurality of software vulnerabilities, a second prediction of whether an exploit that is predicted to be developed for said each software vulnerability is also predicted to be used in an attack.

8. A system comprising:

storage media;

one or more processors; and

one or more programs stored in the storage media and configured for execution by the one or more processors, the one or more programs comprising instructions for:

using one or more first machine learning techniques to generate a first prediction model based on first training data, wherein the first prediction model enables determining a likelihood of whether exploits to be developed for software vulnerabilities will be used in an attack;

wherein the first training data comprises a first plurality of training instances, and wherein each training instance of the first plurality of training instances comprises one or more features corresponding to a respective software vulnerability;

for each training instance of the first plurality of training instances: using the first prediction model to generate a score, and

adding said each training instance to second training data only if the score is greater than a first threshold value;

wherein the second training data is a strict subset of the first training data; and

using one or more second machine learning techniques to generate a second prediction model based on the second training data, wherein the second prediction model enables determining a likelihood of whether exploits to be developed for software vulnerabilities will be used in an attack.

9. The system of claim 8 , wherein the one or more programs further comprise instructions for:

causing application of the first prediction model to first input data that corresponds to a first plurality of software vulnerabilities, wherein each software vulnerability of the first plurality of software vulnerabilities has yet to have an exploit developed for said each software vulnerability;

receiving, based on the application of the first prediction model to the first input data, first output data that indicates, for each software vulnerability of the first plurality of software vulnerabilities, a prediction of whether an exploit to be developed for said each software vulnerability will be used in an attack;

for each prediction indicated in the first output data, adding, to second input data, a software vulnerability corresponding to said each prediction only if said each prediction is above a second threshold value;

causing application of the second prediction model to the second input data; and

receiving, based on the application of the second prediction model to the second input data, second output data that indicates, for each software vulnerability in the second plurality of software vulnerabilities, a prediction of whether an exploit to be developed for said each software vulnerability will be used in an attack.

10. A method comprising:

using one or more first machine learning techniques to generate a first prediction model based on first training data, wherein the first prediction model enables determining a likelihood of whether exploits will be developed for software vulnerabilities;

wherein the first training data comprises a first plurality of training instances, and wherein each training instance of the first plurality of training instances comprises one or more features corresponding to a respective software vulnerability;

for each training instance of the first plurality of training instances: using the first prediction model to generate a corresponding score, and

adding said each training instance to second training data only if the corresponding score is greater than a first threshold value;

wherein the second training data is a strict subset of the first training data;

using one or more second machine learning techniques to generate a second prediction model based on the second training data, wherein the second prediction model enables determining a likelihood of whether exploits will be developed for software vulnerabilities.

11. The method of claim 10 , wherein for each training instance, the corresponding score indicates a likelihood that an exploit will be developed for a software vulnerability corresponding to said each training instance.

12. The method of claim 11 , wherein the first threshold value is a minimum score for which it is likely that an exploit will be developed for the software vulnerability corresponding to said each training instance.

13. The method of claim 10 , further comprising:

causing application of the first prediction model to first input data that corresponds to a first plurality of software vulnerabilities, wherein each software vulnerability of the first plurality of software vulnerabilities has yet to have an exploit developed for said each software vulnerability;

receiving, based on the application of the first prediction model to the first input data, first output data that indicates, for each software vulnerability of the first plurality of software vulnerabilities, a prediction of whether an exploit will be developed for said each software vulnerability;

for each prediction indicated in the first output data, adding, to second input data, a software vulnerability corresponding to said each prediction only if said each prediction is above a second threshold value;

causing application of the second prediction model to the second input data; and

receiving, based on the application of the second prediction model to the second input data, second output data that indicates, for each software vulnerability in the second plurality of software vulnerabilities, a prediction of whether an exploit will be developed for said each software vulnerability.

14. The method of claim 13 , wherein the first threshold value is different from the second threshold value.

15. The method of claim 13 , further comprising:

causing generation of a third prediction model based on third training data, wherein the third training data is a subset of the first training data, and wherein each training instance of the third training data is predicted to have an exploit developed, wherein the third prediction model enables determining a likelihood of whether exploits that have yet to be developed will be used in an attack;

wherein the third training data comprises a second plurality of training instances, and wherein each training instance of the second plurality of training instances comprises one or more features corresponding to a respective software vulnerability;

for each training instance of the second plurality of training instances:

using the third prediction model to generate a second score, and

adding said each training instance to fourth training data only if the score is greater than a third threshold value;

causing generation of a fourth prediction model based on the fourth training data, wherein the fourth prediction model enables determining a likelihood of whether exploits that have yet to be developed will be used in an attack.

16. The method of claim 15 , further comprising:

causing application of the third prediction model to third input data that corresponds to a third plurality of software vulnerabilities, wherein the third plurality of software vulnerabilities is a subset of the second plurality of software vulnerabilities, and wherein each software vulnerability of the third plurality of software vulnerabilities is predicted to have an exploit developed for said each software vulnerability of the third plurality of software vulnerabilities;

receiving, based on the application of the third prediction model to the third input data, third output data that indicates, for each software vulnerability of the third plurality of software vulnerabilities, a first prediction of whether a corresponding exploit that is predicted to be developed for said each software vulnerability is also predicted to be used in an attack;

causing application of the fourth prediction model to fourth input data that corresponds to a fourth plurality of software vulnerabilities, wherein the fourth plurality of software vulnerabilities is determined based on removing, from the third input data, data that corresponds to a subset of the third plurality of software vulnerabilities, wherein each software vulnerability in the subset of the third plurality of software vulnerabilities is predicted to have an exploit developed for said each software vulnerability in the subset, and wherein the corresponding exploit that is predicted to be developed for said each software vulnerability in the subset is predicted not to be used in an attack; and

receiving, based on the application of the fourth prediction model to the fourth input data, fourth output data that indicates, for each software vulnerability in the fourth plurality of software vulnerabilities, a second prediction of whether an exploit that is predicted to be developed for said each software vulnerability is also predicted to be used in an attack.

17. A method comprising:

using one or more first machine learning techniques to generate a first prediction model based on first training data, wherein the first prediction model enables determining a likelihood of whether exploits to be developed for software vulnerabilities will be used in an attack;

wherein the first training data comprises a first plurality of training instances, and wherein each training instance of the first plurality of training instances comprises one or more features corresponding to a respective software vulnerability;

for each training instance of the first plurality of training instances:

using the first prediction model to generate a score, and

adding said each training instance to second training data only if the score is greater than a first threshold value;

wherein the second training data is a strict subset of the first training data; and

using one or more second machine learning techniques to generate a second prediction model based on the second training data, wherein the second prediction model enables determining a likelihood of whether exploits to be developed for software vulnerabilities will be used in an attack.

18. The method of claim 17 , further comprising:

causing application of the first prediction model to first input data that corresponds to a first plurality of software vulnerabilities, wherein each software vulnerability of the first plurality of software vulnerabilities has yet to have an exploit developed for said each software vulnerability;

receiving, based on the application of the first prediction model to the first input data, first output data that indicates, for each software vulnerability of the first plurality of software vulnerabilities, a prediction of whether an exploit to be developed for said each software vulnerability will be used in an attack;

for each prediction indicated in the first output data, adding, to second input data, a software vulnerability corresponding to said each prediction only if said each prediction is above a second threshold value;

causing application of the second prediction model to the second input data; and

receiving, based on the application of the second prediction model to the second input data, second output data that indicates, for each software vulnerability in the second plurality of software vulnerabilities, a prediction of whether an exploit to be developed for said each software vulnerability will be used in an attack.