Multi-stage training of machine learning models

1. A system comprising:

storage media;

one or more processors; and

one or more programs stored in the storage media and configured for execution by the one or more processors, the one or more programs comprising instructions for:

storing a first prediction model that has been trained based on first training data, wherein the first prediction model enables determining a likelihood of whether exploits will be developed for software vulnerabilities;

storing a second prediction model that has been trained based on second training data consisting of training instances of said first training data which, when used to compute a corresponding score through the first prediction model, produced a score greater than a first threshold value, wherein the second prediction model enables determining a likelihood of whether exploits will be developed for software vulnerabilities;

receiving input data identifying a particular software vulnerability; and

using the second prediction model, computing a likelihood of whether exploits will be developed for the particular software vulnerability.

2. The system of claim 1 , wherein for each training instance, the corresponding score indicates a likelihood that an exploit will be developed for a software vulnerability corresponding to said each training instance.

3. The system of claim 2 , wherein scores below the first threshold value indicate that it is unlikely that an exploit will be developed for a software vulnerability.

4. The system of claim 1 , wherein the one or more programs further comprise instructions for:

causing application of the first prediction model to first input data that corresponds to a first plurality of software vulnerabilities, wherein each software vulnerability of the first plurality of software vulnerabilities has yet to have an exploit developed for said each software vulnerability;

receiving, based on the application of the first prediction model to the first input data, first output data that indicates, for each software vulnerability of the first plurality of software vulnerabilities, a prediction of whether an exploit will be developed for said each software vulnerability;

for each prediction indicated in the first output data, adding, to second input data, a software vulnerability corresponding to said each prediction only if said each prediction is above a second threshold value;

causing application of the second prediction model to the second input data; and

receiving, based on the application of the second prediction model to the second input data, second output data that indicates, for each software vulnerability in the second plurality of software vulnerabilities, a prediction of whether an exploit will be developed for said each software vulnerability.

5. The system of claim 4 , wherein the first threshold value is different from the second threshold value.

6. The system of claim 4 , wherein the one or more programs further comprise instructions for:

storing a third prediction model that has been trained based on third training data, wherein the third training data is a subset of the first training data, and wherein each training instance of the third training data is predicted to have an exploit developed, wherein the third prediction model enables determining a likelihood of whether exploits that have yet to be developed will be used in an attack; and

storing a fourth prediction model that has been trained based on fourth training data consisting of training instances of said third training data which, when used to compute a corresponding score through the third prediction model, produced a score greater than a third threshold value, wherein the fourth prediction model enables determining a likelihood of whether exploits that have yet to be developed will be used in an attack.

7. The system of claim 6 , wherein the one or more programs further comprise instructions for:

causing application of the third prediction model to third input data that corresponds to a third plurality of software vulnerabilities, wherein the third plurality of software vulnerabilities is a subset of the second plurality of software vulnerabilities, and wherein each software vulnerability of the third plurality of software vulnerabilities is predicted to have an exploit developed for said each software vulnerability of the third plurality of software vulnerabilities;

receiving, based on the application of the third prediction model to the third input data, third output data that indicates, for each software vulnerability of the third plurality of software vulnerabilities, a first prediction of whether a corresponding exploit that is predicted to be developed for said each software vulnerability is also predicted to be used in an attack;

causing application of the fourth prediction model to fourth input data that corresponds to a fourth plurality of software vulnerabilities, wherein the fourth plurality of software vulnerabilities is determined based on removing, from the third input data, data that corresponds to a subset of the third plurality of software vulnerabilities, wherein each software vulnerability in the subset of the third plurality of software vulnerabilities is predicted to have an exploit developed for said each software vulnerability in the subset, and wherein the corresponding exploit that is predicted to be developed for said each software vulnerability in the subset is predicted not to be used in an attack; and

receiving, based on the application of the fourth prediction model to the fourth input data, fourth output data that indicates, for each software vulnerability in the fourth plurality of software vulnerabilities, a second prediction of whether the corresponding exploit that is predicted to be developed for said each software vulnerability is also predicted to be used in an attack.

8. A system comprising:

storage media;

one or more processors; and

one or more programs stored in the storage media and configured for execution by the one or more processors, the one or more programs comprising instructions for:

storing a first prediction model that has been trained based on first training data, wherein the first prediction model enables determining a likelihood of whether exploits to be developed for software vulnerabilities will be used in an attack;

storing a second prediction model that has been trained based on second training data consisting of training instances of said first training data which, when used to compute a corresponding score through the first prediction model, produced a score greater than a first threshold value, wherein the second prediction model enables determining a likelihood of whether exploits to be developed for software vulnerabilities will be used in an attack;

receiving input data identifying a particular software vulnerability; and

using the second prediction model, computing a likelihood of whether exploits to be developed for the particular software vulnerability will be used in an attack.

9. The system of claim 8 , wherein the one or more programs further comprise instructions for:

causing application of the first prediction model to first input data that corresponds to a first plurality of software vulnerabilities, wherein each software vulnerability of the first plurality of software vulnerabilities has yet to have an exploit developed for said each software vulnerability;

receiving, based on the application of the first prediction model to the first input data, first output data that indicates, for each software vulnerability of the first plurality of software vulnerabilities, a prediction of whether an exploit to be developed for said each software vulnerability will be used in an attack;

for each prediction indicated in the first output data, adding, to second input data, a software vulnerability corresponding to said each prediction only if said each prediction is above a second threshold value;

causing application of the second prediction model to the second input data; and

receiving, based on the application of the second prediction model to the second input data, second output data that indicates, for each software vulnerability in the second plurality of software vulnerabilities, a prediction of whether an exploit to be developed for said each software vulnerability will be used in an attack.

10. A method comprising:

storing a first prediction model that has been trained based on first training data, wherein the first prediction model enables determining a likelihood of whether exploits will be developed for software vulnerabilities;

storing a second prediction model that has been trained based on second training data consisting of training instances of said first training data which, when used to compute a corresponding score through the first prediction model, produced a score greater than a first threshold value, wherein the second prediction model enables determining a likelihood of whether exploits will be developed for software vulnerabilities;

receiving input data identifying a particular software vulnerability; and

using the second prediction model, computing a likelihood of whether exploits will be developed for the particular software vulnerability.

11. The method of claim 10 , wherein for each training instance, the corresponding score indicates a likelihood that an exploit will be developed for a software vulnerability corresponding to said each training instance.

12. The method of claim 11 , wherein the first threshold value is a minimum score for which it is likely that an exploit will be developed for the software vulnerability corresponding to said each training instance.

13. The method of claim 10 , further comprising:

causing application of the first prediction model to first input data that corresponds to a first plurality of software vulnerabilities, wherein each software vulnerability of the first plurality of software vulnerabilities has yet to have an exploit developed for said each software vulnerability;

receiving, based on the application of the first prediction model to the first input data, first output data that indicates, for each software vulnerability of the first plurality of software vulnerabilities, a prediction of whether an exploit will be developed for said each software vulnerability;

for each prediction indicated in the first output data, adding, to second input data, a software vulnerability corresponding to said each prediction only if said each prediction is above a second threshold value;

causing application of the second prediction model to the second input data; and

receiving, based on the application of the second prediction model to the second input data, second output data that indicates, for each software vulnerability in the second plurality of software vulnerabilities, a prediction of whether an exploit will be developed for said each software vulnerability.

14. The method of claim 13 , wherein the first threshold value is different from the second threshold value.

15. The method of claim 13 , further comprising:

storing a third prediction model that has been trained based on third training data, wherein the third training data is a subset of the first training data, and wherein each training instance of the third training data is predicted to have an exploit developed, wherein the third prediction model enables determining a likelihood of whether exploits that have yet to be developed will be used in an attack; and

storing a fourth prediction model that has been trained based on fourth training data consisting of training instances of said third training data which, when used to compute a corresponding score through the third prediction model, produced a score greater than a third threshold value, wherein the fourth prediction model enables determining a likelihood of whether exploits that have yet to be developed will be used in an attack.

16. The method of claim 15 , further comprising:

causing application of the third prediction model to third input data that corresponds to a third plurality of software vulnerabilities, wherein the third plurality of software vulnerabilities is a subset of the second plurality of software vulnerabilities, and wherein each software vulnerability of the third plurality of software vulnerabilities is predicted to have an exploit developed for said each software vulnerability of the third plurality of software vulnerabilities;

receiving, based on the application of the third prediction model to the third input data, third output data that indicates, for each software vulnerability of the third plurality of software vulnerabilities, a first prediction of whether a corresponding exploit that is predicted to be developed for said each software vulnerability is also predicted to be used in an attack;

causing application of the fourth prediction model to fourth input data that corresponds to a fourth plurality of software vulnerabilities, wherein the fourth plurality of software vulnerabilities is determined based on removing, from the third input data, data that corresponds to a subset of the third plurality of software vulnerabilities, wherein each software vulnerability in the subset of the third plurality of software vulnerabilities is predicted to have an exploit developed for said each software vulnerability in the subset, and wherein the corresponding exploit that is predicted to be developed for said each software vulnerability in the subset is predicted not to be used in an attack; and

receiving, based on the application of the fourth prediction model to the fourth input data, fourth output data that indicates, for each software vulnerability in the fourth plurality of software vulnerabilities, a second prediction of whether the corresponding exploit that is predicted to be developed for said each software vulnerability is also predicted to be used in an attack.

17. A method comprising:

storing a first prediction model that has been trained based on first training data, wherein the first prediction model enables determining a likelihood of whether exploits to be developed for software vulnerabilities will be used in an attack;

storing a second prediction model that has been trained based on second training data consisting of training instances of said first training data which, when used to compute a corresponding score through the first prediction model, produced a score greater than a first threshold value, wherein the second prediction model enables determining a likelihood of whether exploits to be developed for software vulnerabilities will be used in an attack;

receiving input data identifying a particular software vulnerability; and

using the second prediction model, computing a likelihood of whether exploits to be developed for the particular software vulnerability will be used in an attack.

18. The method of claim 17 , further comprising:

causing application of the first prediction model to first input data that corresponds to a first plurality of software vulnerabilities, wherein each software vulnerability of the first plurality of software vulnerabilities has yet to have an exploit developed for said each software vulnerability;

receiving, based on the application of the first prediction model to the first input data, first output data that indicates, for each software vulnerability of the first plurality of software vulnerabilities, a prediction of whether an exploit to be developed for said each software vulnerability will be used in an attack;

for each prediction indicated in the first output data, adding, to second input data, a software vulnerability corresponding to said each prediction only if said each prediction is above a second threshold value;

causing application of the second prediction model to the second input data; and

receiving, based on the application of the second prediction model to the second input data, second output data that indicates, for each software vulnerability in the second plurality of software vulnerabilities, a prediction of whether an exploit to be developed for said each software vulnerability will be used in an attack.