Open source vulnerability prediction with machine learning ensemble
A system to create a stacked classifier model combination or classifier ensemble has been designed for identification of undisclosed flaws in software components on a large-scale. This classifier ensemble is capable of at least a 54.55% improvement in precision. The system uses a K-folding cross validation algorithm to partition a sample dataset and then train and test a set of N classifiers with the dataset folds. At each test iteration, trained models of the set of classifiers generate probabilities that a sample has a flaw, resulting in a set of N probabilities or predictions for each sample in the test data. With a sample size of S, the system passes the S sets of N predictions to a logistic regressor along with “ground truth” for the sample dataset to train a logistic regression model. The trained classifiers and the logistic regression model are stored as the classifier ensemble.
1. A method comprising:
training and testing a set of classifiers with k-fold cross validation on a first dataset comprising labels and vectors representing software components,
wherein each label indicates whether a respective one of the software components has a vulnerability or flaw,
wherein a set of probability values generated for each of the vectors by the set of classifiers from the testing is stored;
training a logistic regression model with the stored sets of probability values and the labels; and
combining the trained set of classifiers and the trained logistic regression model to generate an ensemble that indicates whether a software component has a flaw.
2. The method of claim 1 further comprising, for each of the software components, extracting tokens from a software development artifact to generate the vector of the vectors that represents the software component.
3. The method of claim 1 , wherein extracting tokens comprises extracting natural language tokens and programming specific tokens.
4. The method of claim 1 further comprising associating the labels with the vectors based on knowledge of whether the represented software components have vulnerabilities or flaws.
5. The method of claim 1 further comprising collecting data about open source software components, wherein the first dataset is based on the collected data.
6. The method of claim 5 , wherein collecting the data comprises collecting data from commit and bug report repositories.
7. The method of claim 1 , wherein combining to generate the ensemble comprises combining the trained set of classifiers with the trained logistic regression model for the trained set of classifiers to output probabilities for a vector representing a software component and the trained logistic regression model to output a vulnerability probability or vulnerability indicator based on fitting a linear combination of the output probabilities to the trained logistic regression model.
8. The method of claim 1 further comprising:
based on a vector input to the ensemble, generating from the trained set of classifiers a first set of probabilities that the software component represented by the vector has a vulnerability and fitting a linear combination of the first set of probabilities to the trained logistic regression model; and
generating a probability that the vector has a vulnerability based on the fitting.
9. The method of claim 8 further comprising updating a database with an artifact that is a basis for the vector in association with the probability.
10. A non-transitory, computer-readable medium having instructions stored thereon that are executable by a computing device to perform operations comprising:
training and testing a set of classifiers with k-fold cross validation,
wherein the training is with a plurality of vectors each comprising text based tokens extracted from software development artifacts of software components,
wherein the testing is with a plurality of labels that each indicate whether a respective one of the software components has a vulnerability or flaw,
wherein the testing generates multiple sets of probability values that comprise a probability value from each classifier for each of the plurality of vectors;
training a logistic regression model with the multiple sets of probability values and the labels; and
combining the trained set of classifiers and the trained logistic regression model to generate an ensemble that indicates whether a software component has a vulnerability or flaw.
11. The non-transitory, computer-readable medium of claim 10 , wherein the operations further comprise, for each of the software development artifacts, invoking a natural language processor to extract text based tokens to generate the corresponding one of the plurality of vectors.
12. The non-transitory, computer-readable medium of claim 11 , wherein a first of the text based tokens extracted from a software artifact identifies a software component.
13. The non-transitory, computer-readable medium of claim 10 , wherein the text based tokens comprise natural language tokens and programming specific tokens.
14. The non-transitory, computer-readable medium of claim 10 , wherein the operations further comprise collecting the software development artifacts.
15. The non-transitory, computer-readable medium of claim 10 , wherein combining to generate the ensemble comprises combining the trained set of classifiers with the trained logistic regression model for the trained set of classifiers to output probabilities for a vector derived from a software development artifact and the trained logistic regression model to output a vulnerability probability or vulnerability indicator based on fitting a linear combination of the output probabilities to the trained logistic regression model.
16. An apparatus comprising:
a processor; and
a non-transitory machine-readable medium comprising program code stored therein, the program code executable by the processor to cause the apparatus to,
invoke k-fold cross validation to train and test a set of classifiers and program code to train a logistic regression model with probability vectors from the set of classifiers generated from the k-fold cross validation;
input a software development artifact derived vector to an ensemble of the trained set of classifiers and the trained logistic regression model,
wherein the trained set of classifiers generates a set of probabilities that a software component identified by the software development artifact from which the vector is derived has a vulnerability or flaw,
wherein the ensemble generates a probability that the software component has a vulnerability or flaw based on fitting a linear combination of the set of probabilities to the trained logistic regression model; and
update a database with the probability generated from the ensemble in association with identity of the software component.
17. The apparatus of claim 16 , wherein the program code to update the database with the probability in association with the software component identity comprises the program code to update the database with the probability and at least one of the software development artifact and the software development artifact derived vector.
18. The apparatus of claim 16 , wherein the non-transitory machine-readable medium further comprises program code to derive the vector from the software development artifact.
19. The apparatus of claim 18 , wherein the program code to derive the vector comprises program code to extract tokens from the software development artifact with a natural language processor that has been trained to extract natural language tokens related to vulnerabilities or flaws and programming language specific tokens related to vulnerabilities or flaws.