Method and system for reducing false positives in static source code analysis reports using machine learning and classification techniques
This invention is a computer-implemented method and system of using a secondary classification algorithm after using a primary source code vulnerability scanning tool to more accurately label true and false vulnerabilities in source code. The method and system use machine learning within a 10% dataset to develop a classifier model algorithm. A selection process identifies the most important features utilized in the algorithm to detect and distinguish the true and false positive findings of the static code analysis results. A personal identifier is used as a critical feature for the classification. The model is validated by experimentation and comparison against thirteen existing classifiers.
1. A computer-implemented method for labeling true and false vulnerabilities in software code after static code analysis, comprising:
performing a vulnerability scan of static code using one or more open source or proprietary scan tools, to obtain a scan tool vulnerabilities report containing a master data set of labeled vulnerabilities comprising true positive vulnerabilities and false positive vulnerabilities,
the static code selected from source code, binaries, and bytecode, and
aggregating the scan tool vulnerabilities report results into a comma separated values (CSV) file of potential vulnerabilities,
each labeled vulnerability selected from the group consisting of author/personal identifier (ID), severity, common weakness and enumerations (CWE), Subtype, Rule, description, Tool, vulnerability location, and Path;
Creating a training data subset of the master data set from the scan tool vulnerabilities report using machine learning software Waikato Environment for Knowledge Analysis (WEKA) and applying a stratified sampling technique to obtain the training data subset, wherein the training data subset is a 10%-30% data subset of the master data set;
Generate a software assurance personal identifier/vulnerability identification system (SAPI/VISYS) classifier algorithm from the labeled vulnerabilities of the 10%-30% data subset, the SAPI/VISYS classifier algorithm having a formula,
SAPI( Fi )=(( R ( C )+ R ( P )+ R ( S )+ R ( A )+ R ( CA )+ R ( CPA )+ R ( CAS )/ k+R ( CPA )+ R ( CAS ))/ a,
where C is the CWE vulnerability, S is the CWE subtype from part of the description of a function or parameter name, A is the author, P is the path, CA is the combination of author and CWE, CP is the combination of path and CWE, CPA is the combination of CWE, path and author, CAS is the combination of CWE, author and subtype, k is the count of the types of vulnerabilities (CWE, path, subtype, author), “a” is the count of the number of times the combination of CPA and CAS appears in the 10%-30% data subset;
where the algorithm calculates for each vulnerability a probability of being a true positive vulnerability, and uses a 20.3%-50% observed false positive (OFP) threshold to separate a true positive vulnerability from a false positive vulnerability;
Performing a SAPI/VISYS classifier algorithm scan of the master data set to obtain a SAPI/VISYS classifier algorithm scan tool vulnerabilities report containing a revised master data set of re-labeled vulnerabilities comprising true positive vulnerabilities and 90-95% less false positive vulnerabilities, compared to the original master data set;
wherein the method is performed by one or more computing devices.
2. A system for labeling true and false vulnerabilities in software code after static code analysis comprising:
one or more processors;
one or more storage media storing instructions which, when executed by the one or more processors are configured to provide
one or more open source or proprietary scan tools configured to perform a vulnerability scan of static code to obtain a scan tool vulnerabilities report containing a master data set of labeled vulnerabilities comprising true positive vulnerabilities and false positive vulnerabilities, the static code selected from source code, binaries, and bytecode, and
a comma separated values (CSV) file of potential vulnerabilities aggregated from the scan tool vulnerabilities report results,
wherein each labeled vulnerability selected from the group consisting of author/personal identifier (ID), severity, common weakness and enumerations (CWE), Subtype, Rule, description, Tool, vulnerability location, and Path;
a training data subset of the master data set created from the scan tool vulnerabilities report using machine learning software Waikato Environment for Knowledge Analysis (WEKA) and a stratified sampling technique to obtain the training data subset, wherein the training data subset is a 10%-30% data subset of the master data set;
a software assurance personal identifier/vulnerability identification system (SAPI/VISYS) classifier algorithm generated from the labeled vulnerabilities of the 10%-30% data subset, the SAPI/VISYS classifier algorithm having a formula,
SAPI( Fi )=(( R ( C )+ R ( P )+ R ( S )+ R ( A )+ R ( CA )+ R ( CPA )+ R ( CAS )/ k+R ( CPA )+ R ( CAS ))/ a,
where C is the CWE vulnerability, S is the CWE subtype from part of the description of a function or parameter name, A is the author, P is the path, CA is the combination of author and CWE, CP is the combination of path and CWE, CPA is the combination of CWE, path and author, CAS is the combination of CWE, author and subtype, k is the count of the types of vulnerabilities (CWE, path, subtype, author), “a” is the count of the number of times the combination of CPA and CAS appears in the 10%-30% data subset;
where the algorithm calculates for each vulnerability a probability of being a true positive vulnerability, and uses a 20.3%-50% observed false positive (OFP) threshold to separate a true positive vulnerability from a false positive vulnerability;
the SAPI/VISYS classifier algorithm configured to scan the master data set to obtain a SAPI/VISYS classifier algorithm scan tool vulnerabilities report containing a revised master data set of re-labeled vulnerabilities comprising true positive vulnerabilities and 90-95% less false positive vulnerabilities, compared to the original master data set.
3. The method of claim 1 , where the author/personal identifier (ID) vulnerability is “no author” and is replaced with a file properties vulnerability in the SAPI/VISYS formula, where C is the CWE vulnerability, S is the CWE subtype, P is the location/path, FN is the filename where the vulnerability resides, FL is the length of the file where the vulnerability resides, LW is the last write information (date) of the file where the vulnerability resides, CP is the combination of CWE and Path (location), CL is the combination of CWE and File length, CPL is the combination of CWE, Path and file length, CA is the combination of author and CWE and A (filename, file length, last write), CPA is the combination of CWE, Path and A (filename, file length, last write), CAS is the combination of CWE, Subtype and A (filename, file length, last write), CFIS is the combination of CWE, File length and sub type, and
wherein an extra feature called “bias” is added which is the observed true positive (OTP) ratio of all the findings of the training dataset, and
wherein C OTP is calculated as R(C), and
wherein all the above ratios are averaged together.
4. The method of claim 1 , where the vulnerabilities are in the SAPI/VISYS formula are weighted.
5. The method of claim 1 , wherein the accuracy, precision, and recall of the SAPI/VISYS formula has a 41% decrease in false positives compared to stochastic gradient descent (SGD).