IP Library Granted Patent US 12,386,978
Granted Patent B2
US 12,386,978 · App. 18/079,665 · Granted Aug 12, 2025

Directed fuzzing for vulnerability detection

Inventors: Tom Ganz (Karlsruhe, DE); Martin Haerterich (Wiesloch, DE); Philipp Rall (Darmstadt, DE)
Assignee: SAP SE
G06F21/577G06F11/3698G06F21/563G06F2221/033
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,386,978
App. No.
18/079,665
Granted
Aug 12, 2025
Kind
B2
Abstract

Applications may contain vulnerabilities to attack via malicious inputs. Machine-learning models may be trained to detect these vulnerabilities by accepting source code as input and outputting a probability that each of a set of vulnerabilities exists in the source code. Explanation methods may identify one or more locations within the source code that are likely to cause the vulnerability. Directed fuzzing provides a range of inputs to source code. The inputs that cause the source code to fail are detected and the portions of the source code that were vulnerable are identified. The results of the directed fuzzing are used to select between explanations generated by multiple explanation methods, to provide additional training data to a machine-learning model, to provide additional training data to an explanation method, or any suitable combination thereof.

Claims (77)

1. A system comprising:

a memory that stores instructions; and

one or more processors configured by the instructions to perform operations comprising:

training a machine-learning model to detect whether source code contains a vulnerability;

determining, using the trained machine-learning model, whether the vulnerability exists in a source code;

identifying, using a first explanation method, one or more first target locations corresponding to the vulnerability within the source code;

identifying, using a second explanation method, one or more second target locations corresponding to the vulnerability within the source code;

using guided fuzzing, providing input to the source code targeting the first target locations to empirically measure whether the vulnerability exists at the first target locations by running the source code in a debugger with breakpoints set at the one or more first target locations;

using guided fuzzing, providing input to the source code targeting the second target locations to empirically measure whether the vulnerability exists at the second target locations by running the source code in the debugger with breakpoints set at the one or more second target locations;

based on results from the guided fuzzing, selecting between the first target locations and the second target locations based on mean breakpoint hits when running the source code in the debugger; and

indicating, on a user interface, the selected target locations.

2. The system of claim 1 , wherein the identifying of the one or more first target locations comprises:

generating, by the one or more processors, based on the source code and the determination of whether the vulnerability exists in the source code, a map that attributes numerical relevance scores to locations in the source code;

sorting the locations of the map by relevance; and

converting the locations to line numbers in the source code.

3. The system of claim 1 , wherein the empirical measuring of whether the vulnerability exists at the first target locations or the second target locations is further based on a mean crash distance between the one or more target locations and locations where the source code crashes.

4. The system of claim 1 , wherein the identifying of the one or more first target locations in the source code that may cause the vulnerability comprises:

converting the source code to a graph representation;

providing the graph representation to the first explanation method;

receiving one or more graph locations from the first explanation method; and

converting the one or more graph locations to one or more line numbers in the source code.

5. The system of claim 1 , wherein the operations further comprise:

modifying the explanation method based on the source code and the empirical measuring of whether the vulnerability exists at the first target locations or the second target locations.

6. The system of claim 1 , wherein the determining, using the trained machine-learning model, whether the vulnerability exists in the source code comprises:

providing the source code as input to the trained machine-learning model; and

receiving, as output from the trained machine-learning model, a vector comprising a plurality of elements, each element indicating a probability of a corresponding vulnerability existing in the input.

7. The system of claim 1 , wherein the providing input to the source code targeting the first target locations to empirically measure whether the vulnerability exists at the first target locations comprises executing the source code multiple times with different inputs generated by a fuzzing module.

8. A method comprising:

training a machine-learning model to detect whether source code contains a vulnerability;

determining, by one or more processors using the trained machine-learning model, whether the vulnerability exists in a source code;

identifying, using a first explanation method, one or more first target locations corresponding to the vulnerability within the source code;

identifying, using a second explanation method, one or more second target locations corresponding to the vulnerability within the source code;

using guided fuzzing, providing input to the source code targeting the first target locations to empirically measure whether the vulnerability exists at the first target locations by running the source code in a debugger with breakpoints set at the one or more first target locations;

using guided fuzzing, providing input to the source code targeting the second target locations to empirically measure whether the vulnerability exists at the second target locations by running the source code in the debugger with breakpoints set at the one or more second target locations;

based on results from the guided fuzzing, selecting between the first target locations and the second target locations based on mean breakpoint hits when running the source code in the debugger; and

indicating, on a user interface, the selected target locations.

9. The method of claim 8 , wherein the identifying of the one or more first target locations comprises:

generating, by the one or more processors, based on the source code and the determination of whether the vulnerability exists in the source code, a map that attributes numerical relevance scores to locations in the source code;

sorting the locations of the map by relevance; and

converting the locations to line numbers in the source code.

10. The method of claim 8 , wherein the empirical measuring of whether the vulnerability exists at the first target locations or the second target locations is further based on a mean crash distance between the one or more target locations and locations where the source code crashes.

11. The method of claim 8 , wherein the identifying of the one or more first target locations in the source code that may cause the vulnerability comprises:

converting the source code to a graph representation;

providing the graph representation to the first explanation method;

receiving one or more graph locations from the first explanation method; and

converting the one or more graph locations to one or more line numbers in the source code.

12. The method of claim 8 , further comprising:

modifying the explanation method based on the source code and the empirical measuring of whether the vulnerability exists at the first target locations or the second target locations.

13. The method of claim 8 , wherein the determining, using the trained machine-learning model, whether the vulnerability exists in the source code comprises:

providing the source code as input to the trained machine-learning model; and

receiving, as output from the trained machine-learning model, a vector comprising a plurality of elements, each element indicating a probability of a corresponding vulnerability existing in the input.

14. The method of claim 8 , wherein the providing input to the source code targeting the first target locations to empirically measure whether the vulnerability exists at the first target locations comprises executing the source code multiple times with different inputs generated by a fuzzing module.

15. A non-transitory computer-readable medium that stores instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:

training a machine-learning model to detect whether source code contains a vulnerability;

determining, using the trained machine-learning model, whether the vulnerability exists in a source code;

identifying, using a first explanation method, one or more first target locations corresponding to the vulnerability within the source code;

identifying, using a second explanation method, one or more second target locations corresponding to the vulnerability within the source code;

using guided fuzzing, providing input to the source code targeting the first target locations to empirically measure whether the vulnerability exists at the first target locations by running the source code in a debugger with breakpoints set at the one or more first target locations;

using guided fuzzing, providing input to the source code targeting the second target locations to empirically measure whether the vulnerability exists at the second target locations by running the source code in the debugger with breakpoints set at the one or more second target locations;

based on results from the guided fuzzing, selecting between the first target locations and the second target locations based on mean breakpoint hits when running the source code in the debugger; and

indicating, on a user interface, the selected target locations.

16. The non-transitory computer-readable medium of claim 15 , wherein the identifying of the one or more first target locations comprises:

generating, by the one or more processors, based on the source code and the determination of whether the vulnerability exists in the source code, a map that attributes numerical relevance scores to locations in the source code;

sorting the locations of the map by relevance; and

converting the locations to line numbers in the source code.

17. The non-transitory computer-readable medium of claim 15 , wherein the empirical measuring of whether the vulnerability exists at the first target locations or the second target locations is further based on a mean crash distance between the one or more target locations and locations where the source code crashes.

18. The non-transitory computer-readable medium of claim 15 , wherein the operations further comprise:

modifying the explanation method based on the source code and the empirical measuring of whether the vulnerability exists at the first target locations or the second target locations.

19. The non-transitory computer-readable medium of claim 15 , wherein the identifying of the one or more first target locations in the source code that may cause the vulnerability comprises:

converting the source code to a graph representation;

providing the graph representation to the first explanation method;

receiving one or more graph locations from the first explanation method; and

converting the one or more graph locations to one or more line numbers in the source code.

20. The non-transitory computer-readable medium of claim 15 , wherein the determining, using the trained machine-learning model, whether the vulnerability exists in the source code comprises:

providing the source code as input to the trained machine-learning model; and

receiving, as output from the trained machine-learning model, a vector comprising a plurality of elements, each element indicating a probability of a corresponding vulnerability existing in the input.

21. The non-transitory computer-readable medium of claim 15 , wherein the providing input to the source code targeting the first target locations to empirically measure whether the vulnerability exists at the first target locations comprises executing the source code multiple times with different inputs generated by a fuzzing module.

Assignments (2)
CORRECTIVE ASSIGNMENT TO CORRECT THE ASSIGNEENAME TO SAP SE PREVIOUSLY RECORDED ON REEL 062060 FRAME 0163. ASSIGNOR(S) HEREBY CONFIRMS THE ASSIGNMENT. Recorded May 22, 2023
From: GANZ, TOM; HAERTERICH, MARTIN; RALL, PHILIPP
To: SAP SE
Reel/Frame 064105/0143 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 12, 2022
From: GANZ, TOM; HAERTERICH, MARTIN; RALL, PHILIPP
To: SE, SAP
Reel/Frame 062060/0163 →
Continuity (2)
Provisional Application 63419079 · Oct 25, 2022
Related Publication 20240184892A1 · Jun 6, 2024
References Cited (60)
US 8719791B1 · MacPherson · 2014 [cited by examiner]
US 10380350B1 · Ben Porath · 2019 [cited by examiner]
US 11816221B2 · Singh · 2023 [cited by applicant]
US 11882143B1 · Garcia Correa · 2024 [cited by applicant]
US 12153684B2 · Chan et al. · 2024 [cited by applicant]
US 20060288332A1 · Sagar · 2006 [cited by examiner]
US 20120204062A1 · Erickson · 2012 [cited by examiner]
US 20150095894A1 · Tripp · 2015 [cited by examiner]
US 20180232523A1 · Copty · 2018 [cited by examiner]
US 20190340103A1 · Nelson et al. · 2019 [cited by applicant]
US 20190362077A1 · Schornack · 2019 [cited by examiner]
US 20190370473A1 · Matrosov · 2019 [cited by examiner]
US 20200394311A1 · Li · 2020 [cited by examiner]
US 20200401504A1 · Sommers et al. · 2020 [cited by applicant]
US 20210056211A1 · Olson et al. · 2021 [cited by applicant]
US 20210279338A1 · Bowman et al. · 2021 [cited by applicant]
US 20220253533A1 · Choi · 2022 [cited by examiner]
US 20230104814A1 · Johnson et al. · 2023 [cited by applicant]
US 20230146443A1 · Huth · 2023 [cited by examiner]
US 20230169164A1 · Bishop, III et al. · 2023 [cited by applicant]
US 20230169177A1 · Bishop, III et al. · 2023 [cited by applicant]
US 20230281317A1 · Clement · 2023 [cited by examiner]
US 20230325301A1 · Feng · 2023 [cited by examiner]
US 20240028740A1 · Chan et al. · 2024 [cited by applicant]
US 20240045971A1 · Ben Salem et al. · 2024 [cited by applicant]
US 20240184891A1 · Ganz et al. · 2024 [cited by applicant]
US 20240281528A1 · De Macedo · 2024 [cited by applicant]
“U.S. Appl. No. 18/079,611, Non Final Office Action mailed Dec. 4, 2024”, 9 pgs. [cited by applicant]
Allamanis, Miltiadis, “Learning to represent programs with graphs”, ICLR, [Online]. Retrieved from the Internet: URL: https: arxiv.org pdf 1711.00740.pdf, (5 4 18), 17 pgs. [cited by applicant]
Arusoaie, Andrei, “A Comparison of Open-Source Static Analysis Tools for Vulnerability Detection in C C++ Code”, 19th International Symposium on Symbolic and Numeric Algorithms for Scientific Computing (SYNASC), (2017),… [cited by applicant]
Ben-Nun, “Neural Code Comprehension: A Learnable Representation of Code Semantics”, [Online] Retrieved from the internet:https: arxiv.org abs 1806.07336, (2018), 1-17. [cited by applicant]
Bohme, Marcel, “Directed Greybox Fuzzing”, Session K2: Fuzzing Finer and Faster, CCS'17, Dallas, TX, USA, (Oct. 30-Nov. 3, 2017), 16 pgs. [cited by applicant]
Burkart, Nadia, “A Survey on the Explainability of Supervised Machine Learning”, Journal of Artificial Intelligence Research 70, (Jan. 2021), 245-317. [cited by applicant]
Chakraborty, Saikat, “Deep Learning based Vulnerability Detection: Are We There Yet?”, IEEE Transactions on Software Engineering, vol. TBD, (Sep. 3, 2020), 19 pgs. [cited by applicant]
Chen, Hongxu, “Hawkeye: Towards a Desired Directed Grey-box Fuzzer”, [Online] Retrieved from the internet: https: doi.org 10.1145 3243734.3243849, (Oct. 15-19, 2018), 14 pgs. [cited by applicant]
Ferrante, J., “The Program Dependence Graph and Its Use in Optimization”, ACM Transactions on Programming Languages and Systems, 9, (Jul. 1987), 319-349. [cited by applicant]
Ganz, Tom, “Explaining Graph Neural Networks for Vulnerability Discovery”, Session 2B: Machine Learning for Cybersecurity, AlSec '21, Virtual Event, Republic of Korea, (Nov. 15, 2021), 12 pgs. [cited by applicant]
Griecox, Gustavo, “Toward large-scale vulnerability discovery using Machine Learning”, 12 pgs. [cited by applicant]
Guo, Wenbo, “LEMNA: Explaining Deep Learning based Security Applications”, Session 2D: ML 2, CCS'18, Toronto, ON, Canada, (Oct. 15-19, 2018), 16 pgs. [cited by applicant]
Klees, George, “Evaluating Fuzz Testing”, Session 10D: VulnDet 2 + Side Channels 2, CCS'18, Toronto, ON, Canada, (Oct. 15-19, 2018), 16 pgs. [cited by applicant]
Li, Zhen, “SySeVR: A Framework for Using Deep Learning to Detect Software Vulnerabilities”, IEEE Transactions on Dependable and Secure Computing, (Jan. 12, 2021), 15 pgs. [cited by applicant]
Li, Zhen, “VulDeePecker: A Deep Learning-Based System for Vulnerability Detection”, Network and Distributed Systems Security (NDSS) Symposium, [Online] Retrieved from the internet:http: dx.doi.org 10.14722 ndss.2018.231… [cited by applicant]
Luo, Dongsheng, “Parameterized Explainer for Graph Neural Network”, 34th Conference on Neural Information Processing Systems (NeurIPS 2020), Vancouver, Canada, (2020), 12 pgs. [cited by applicant]
Manes, Valentin J.M., “The Art, Science, and Engineering of Fuzzing: A Survey”, (Apr. 8, 2019), 21 pgs. [cited by applicant]
Mazuera-Rozo, Alejandro, “Shallow or Deep? An Empirical Study on Detecting Vulnerabilities using Deep Learning”, Both authors contributed equally to this manuscript, (Mar. 22, 2021), 12 pgs. [cited by applicant]
Mikolov, Tomas, “Distributed Representations of Words and Phrases and their Compositionality”, arXiv:1310.4546v1 [cs.CL], (Oct. 16, 2013), 9 pgs. [cited by applicant]
Osterlund, Sebastian, “ParmeSan: Sanitizer-guided Greybox Fuzzing”, Open access to the Proceedings of the 29th USENIX Security Symposiumis sponsored by USENIX, [Online] Retrieved from the internet:https: www.usenix. org… [cited by applicant]
Russell, Rebecca L, “Automated Vulnerability Detection in Source Code Using Deep Representation Learning”, 17th IEEE International Conference on Machine Learning and Applications, (2018), 7 pgs. [cited by applicant]
Sanchez-Lengeling, Benjamin, “Evaluating Attribution for Graph Neural Networks”, 34th Conference on Neural Information Processing Systems (NeurIPS 2020), Vancouver, Canada, (2020), 13 pgs. [cited by applicant]
Selvaraju, Ramprassaath R., “Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization”, Proceedings of the IEEE International Conference on Computer Vision (ICCV), (2017), 618-626. [cited by applicant]
Smilkov, Daniel, “SmoothGrad: removing noise by adding noise”, Sharper sensitivity maps: removing noise by adding noise, (Jun. 12, 2017), 10 pgs. [cited by applicant]
Warnecke, Alexander, “Evaluating Explanation Methods for Deep Learning in Security”, IEEE European Symposium on Security and Privacy (EuroSandP), (2020), 17 pgs. [cited by applicant]
Wu, Zonghan, “A Comprehensive Survey on Graph Neural Networks”, Journal of Latex Class Files, vol. XX, No. XX, (Dec. 4, 2019), 22 pgs. [cited by applicant]
Yamaguchi, Fabian, “Modeling and Discovering Vulnerabilities with Code Property Graphs”, IEEE Symposium on Security and Privacy, [Online]. Retrieved from the Internet: URL: https: comsecuris.com papers 06956589.pdf, (20… [cited by applicant]
Ying, Rex, “GNNExplainer: Generating Explanations for Graph Neural Networks”, 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada., (2019), 12 pgs. [cited by applicant]
Zhou, Yaqin, “Devign: Effective Vulnerability Identification by Learning Comprehensive Program Semantics via Graph Neural Networks”, 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Ca… [cited by applicant]
Zou, Deqing, “Interpreting Deep Learning-based Vulnerability Detector Predictions Based on Heuristic Searching”, ACM Trans. Softw. Eng. Methodol., vol. 37, No. 4, Article 111, (Aug. 2018), 33 pgs. [cited by applicant]
“U.S. Appl. No. 18/079,611, Examiner Interview Summary mailed Feb. 3, 2025”, 2 pgs. [cited by applicant]
“U.S. Appl. No. 18/079,611, Response filed Mar. 3, 2025 to Non Final Office Action mailed Dec. 4, 2024”, 11 pgs. [cited by applicant]
“U.S. Appl. No. 18/079,611, Final Office Action mailed May 19, 2025”, 11 pgs. [cited by applicant]