IP Library › Granted Patent US 12,462,037
Granted Patent B2
US 12,462,037 · App. 18/079,611 · Granted Nov 4, 2025

Directed fuzzing for vulnerability detection

Inventors: Tom Ganz (Karlsruhe, DE); Martin Haerterich (Wiesloch, DE); Philipp Rall (Darmstadt, DE)
Assignee: SAP SE
G06F21/577G06F11/3698G06F21/563G06F2221/033
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,462,037
App. No.
18/079,611
Granted
Nov 4, 2025
Kind
B2
Abstract

Applications may contain vulnerabilities to attack via malicious inputs. Machine-learning models may be trained to detect these vulnerabilities by accepting source code as input and outputting a probability that each of a set of vulnerabilities exists in the source code. Explanation methods may identify one or more locations within the source code that are likely to cause the vulnerability. Directed fuzzing provides a range of inputs to source code. The inputs that cause the source code to fail are detected and the portions of the source code that were vulnerable are identified. The results of the directed fuzzing are used to select between explanations generated by multiple explanation methods, to provide additional training data to a machine-learning model, to provide additional training data to an explanation method, or any suitable combination thereof.

Claims (48)

1 . A system comprising:

a memory that stores instructions; and

one or more processors configured by the instructions to perform operations comprising:

generating, using a machine-learning model, a probability of a vulnerability existing in source code;

identifying, using one or more explanation methods, one or more locations in the source code that may cause the vulnerability;

based on the probability of the vulnerability existing, using random fuzzing, providing input to the source code to empirically measure whether the vulnerability exists in the source code, wherein the providing of the input to the source code comprises running the source code in a debugger with breakpoints set at the one or more locations; and

training the machine-learning model based on the source code and the empirical measurement.

2 . The system of claim 1 , wherein the operations further comprise:

generating, based on the source code and the probability, a map that attributes numerical relevance scores to locations in the source code; and

sorting the locations of the map by relevance.

3 . The system of claim 2 , wherein the providing of the input to the source code using random fuzzing comprises using guided fuzzing with the locations as targets to provide input to the source code.

4 . The system of claim 1 , wherein the empirical measurement is based on mean breakpoint hits when running the source code in the debugger.

5 . The system of claim 1 , wherein the empirical measurement is based on a mean crash distance between the one or more locations and locations where the source code crashes.

6 . The system of claim 1 , wherein the identifying of the one or more locations in the source code that may cause the vulnerability comprises:

converting the source code to a graph representation;

providing the graph representation to the one or more explanation methods;

receiving one or more graph locations from the one or more explanation methods; and

converting the one or more graph locations to one or more line numbers in the source code.

7 . A method comprising:

generating, by one or more processors using a machine-learning model, a probability of a vulnerability existing in source code;

identifying, using one or more explanation methods, one or more locations in the source code that may cause the vulnerability;

based on the probability of the vulnerability existing, using random fuzzing, providing input to the source code to empirically measure whether the vulnerability exists in the source code, wherein the providing of the input to the source code comprises running the source code in a debugger with breakpoints set at the one or more locations; and

training, by the one or more processors, the machine-learning model based on the source code and the empirical measurement.

8 . The method of claim 7 , further comprising:

generating, by the one or more processors, based on the source code and the probability, a map that attributes numerical relevance scores to locations in the source code; and

sorting the locations of the map by relevance.

9 . The method of claim 8 , wherein the providing of the input to the source code using random fuzzing comprises using guided fuzzing with the locations as targets to provide input to the source code.

10 . The method of claim 7 , wherein the empirical measurement is based on mean breakpoint hits when running the source code in the debugger.

11 . The method of claim 7 , wherein the empirical measurement is based on a mean crash distance between the one or more locations and locations where the source code crashes.

12 . The method of claim 7 , wherein the identifying of the one or more locations in the source code that may cause the vulnerability comprises:

converting the source code to a graph representation;

providing the graph representation to the one or more explanation methods;

receiving one or more graph locations from the one or more explanation methods; and

converting the one or more graph locations to one or more line numbers in the source code.

13 . A non-transitory computer-readable medium that stores instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:

generating, using a machine-learning model, a probability of a vulnerability existing in source code;

identifying, using one or more explanation methods, one or more locations in the source code that may cause the vulnerability;

based on the probability of the vulnerability existing, using random fuzzing, providing input to the source code to empirically measure whether the vulnerability exists in the source code, wherein the providing of the input to the source code comprises running the source code in a debugger with breakpoints set at the one or more locations; and

training the machine-learning model based on the source code and the empirical measurement.

14 . The non-transitory computer-readable medium of claim 13 , wherein the operations further comprise:

generating, based on the source code and the probability, a map that attributes numerical relevance scores to locations in the source code; and

sorting the locations of the map by relevance.

15 . The non-transitory computer-readable medium of claim 14 , wherein the providing of the input to the source code using random fuzzing comprises using guided fuzzing with the locations as targets to provide input to the source code.

16 . The non-transitory computer-readable medium of claim 13 , wherein the empirical measurement is based on mean breakpoint hits when running the source code in the debugger.

17 . The non-transitory computer-readable medium of claim 13 , wherein the empirical measurement is based on a mean crash distance between the one or more locations and locations where the source code crashes.

18 . The system of claim 5 , wherein the mean crash distance is measured by an average time that elapses from a breakpoint hit to a crash site.

19 . The method of claim 11 , wherein the mean crash distance is measured by an average time that elapses from a breakpoint hit to a crash site.

20 . The non-transitory computer-readable medium of claim 17 , wherein the mean crash distance is measured by an average time that elapses from a breakpoint hit to a crash site.

Assignments (2)
CORRECTIVE ASSIGNMENT TO CORRECT THE ASSIGNEENAME TO SAP SE PREVIOUSLY RECORDED ON REEL 062059 FRAME 0823. ASSIGNOR(S) HEREBY CONFIRMS THE ASSIGNMENT. Recorded May 18, 2023
From: GANZ, TOM; HAERTERICH, MARTIN; RALL, PHILIPP
To: SAP SE
Reel/Frame 063690/0704 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 12, 2022
From: GANZ, TOM; HAERTERICH, MARTIN; RALL, PHILIPP
To: SE, SAP
Reel/Frame 062059/0823 →
Continuity (2)
Provisional Application 63419079 · Oct 25, 2022
Related Publication 20240184891A1 · Jun 6, 2024
References Cited (65)
US 8719791B1 · Macpherson et al. · 2014 [cited by applicant]
US 10380350B1 · Ben Porath · 2019 [cited by examiner]
US 10621361B2 · Holz · 2020 [cited by examiner]
US 10885020B1 · Ablitt · 2021 [cited by examiner]
US 11544626B2 · Adeli-Nadjafi · 2023 [cited by examiner]
US 11564101B2 · Fu · 2023 [cited by examiner]
US 11816102B2 · Parravicini · 2023 [cited by examiner]
US 11816221B2 · Singh · 2023 [cited by examiner]
US 11882143B1 · Garcia Correa · 2024 [cited by examiner]
US 12153684B2 · Chan · 2024 [cited by examiner]
US 20060288332A1 · Sagar et al. · 2006 [cited by applicant]
US 20120204062A1 · Erickson et al. · 2012 [cited by applicant]
US 20150095894A1 · Tripp · 2015 [cited by applicant]
US 20180232523A1 · Copty et al. · 2018 [cited by applicant]
US 20190340103A1 · Nelson · 2019 [cited by examiner]
US 20190362077A1 · Schornack et al. · 2019 [cited by applicant]
US 20190370473A1 · Matrosov et al. · 2019 [cited by applicant]
US 20200394311A1 · Li et al. · 2020 [cited by applicant]
US 20200401504A1 · Sommers · 2020 [cited by examiner]
US 20210056211A1 · Olson · 2021 [cited by examiner]
US 20210279338A1 · Bowman · 2021 [cited by examiner]
US 20220253533A1 · Choi · 2022 [cited by applicant]
US 20230104814A1 · Johnson · 2023 [cited by examiner]
US 20230146443A1 · Huth et al. · 2023 [cited by applicant]
US 20230169164A1 · Bishop, III · 2023 [cited by examiner]
US 20230169177A1 · Bishop, III · 2023 [cited by examiner]
US 20230281317A1 · Clement et al. · 2023 [cited by applicant]
US 20230325301A1 · Feng et al. · 2023 [cited by applicant]
US 20240028740A1 · Chan · 2024 [cited by examiner]
US 20240045971A1 · Ben Salem · 2024 [cited by examiner]
US 20240184892A1 · Ganz et al. · 2024 [cited by applicant]
“U.S. Appl. No. 18/079,665, Final Office Action mailed Mar. 21, 2025”, 29 pgs. [cited by applicant]
“U.S. Appl. No. 18/079,665, Examiner Interview Summary mailed Apr. 22, 2025”, 3 pgs. [cited by applicant]
“U.S. Appl. No. 18/079,665, Non Final Office Action mailed Sep. 12, 2024”, 22 pgs. [cited by applicant]
“U.S. Appl. No. 18/079,665, Examiner Interview Summary mailed Oct. 18, 2024”, 3 pgs. [cited by applicant]
“U.S. Appl. No. 18/079,665, Response filed Dec. 12, 2024 to Non Final Office Action mailed Sep. 12, 2024”, 14 pgs. [cited by applicant]
Allamanis, Miltiadis, “Learning to represent programs with graphs”, ICLR, [Online]. Retrieved from the Internet: URL: https: arxiv.org pdf 1711.00740.pdf, (May 4, 2018), 17 pgs. [cited by applicant]
Arusoaie, Andrei, “A Comparison of Open-Source Static Analysis Tools for Vulnerability Detection in C C++ Code”, 19th International Symposium on Symbolic and Numeric Algorithms for Scientific Computing (SYNASC), (2017),… [cited by applicant]
Ben-Nun, “Neural Code Comprehension: A Learnable Representation of Code Semantics”, [Online] Retrieved from the internet:https: arxiv.org abs 1806.07336, (2018), 1-17. [cited by applicant]
Bohme, Marcel, “Directed Greybox Fuzzing”, Session K2: Fuzzing Finer and Faster, CCS'17, Dallas, TX, USA, (Oct. 30-Nov. 3, 2017), 16 pgs. [cited by applicant]
Burkart, Nadia, “A Survey on the Explainability of Supervised Machine Learning”, Journal of Artificial Intelligence Research 70, (Jan. 2021), 245-317. [cited by applicant]
Chakraborty, Saikat, “Deep Learning based Vulnerability Detection: Are We There Yet?”, IEEE Transactions on Software Engineering, vol. TBD, (Sep. 3, 2020), 19 pgs. [cited by applicant]
Chen, Hongxu, “Hawkeye: Towards a Desired Directed Grey-box Fuzzer”, [Online] Retrieved from the internet: https: doi.org 10.1145 3243734.3243849, (Oct. 15-19, 2018), 14 pgs. [cited by applicant]
Ferrante, J., “The Program Dependence Graph and Its Use in Optimization”, ACM Transactions on Programming Languages and Systems, 9 319-349, (Jul. 1987), 31 pgs. [cited by applicant]
Ganz, Tom, “Explaining Graph Neural Networks for Vulnerability Discovery”, Session 2B: Machine Learning for Cybersecurity, AlSec '21, Virtual Event, Republic of Korea, (Nov. 15, 2021), 12 pgs. [cited by applicant]
Griecox, Gustavo, “Toward large-scale vulnerability discovery using Machine Learning”, 12 pgs. [cited by applicant]
Guo, Wenbo, “LEMNA: Explaining Deep Learning based Security Applications”, Session 2D: ML 2, CCS'18, Toronto, ON, Canada, (Oct. 15-19, 2018), 16 pgs. [cited by applicant]
Klees, George, “Evaluating Fuzz Testing”, Session 10D: VulnDet 2 + Side Channels 2, CCS'18, Toronto, ON, Canada, (Oct. 15-19, 2018), 16 pgs. [cited by applicant]
Li, Zhen, “SySeVR: A Framework for Using Deep Learning to Detect Software Vulnerabilities”, IEEE Transactions on Dependable and Secure Computing, (Jan. 12, 2021), 15 pgs. [cited by applicant]
Li, Zhen, “VulDeePecker: A Deep Learning-Based System for Vulnerability Detection”, Network and Distributed Systems Security (NDSS) Symposium, [Online] Retrieved from the internet:http: dx.doi.org 10.14722ndss.2018.2315… [cited by applicant]
Luo, Dongsheng, “Parameterized Explainer for Graph Neural Network”, 34th Conference on Neural Information Processing Systems (NeurIPS 2020), Vancouver, Canada, (2020), 12 pgs. [cited by applicant]
Manes, Valentin J.M., “The Art, Science, and Engineering of Fuzzing: A Survey”, (Apr. 8, 2019), 21 pgs. [cited by applicant]
Mazuera-Rozo, Alejandro, “Shallow or Deep? An Empirical Study on Detecting Vulnerabilities using Deep Learning”, Both authors contributed equally to this manuscript, (Mar. 22, 2021), 12 pgs. [cited by applicant]
Mikolov, Tomas, “Distributed Representations of Words and Phrases and their Compositionality”, arXiv:1310.4546v1 [cs.CL], (Oct. 16, 2013), 9 pgs. [cited by applicant]
Osterlund, Sebastian, “ParmeSan: Sanitizer-guided Greybox Fuzzing”, Open access to the Proceedings of the 29th USENIX Security Symposiumis sponsored by USENIX, [Online] Retrieved from the internet:https: www.usenix.org … [cited by applicant]
Russell, Rebecca L, “Automated Vulnerability Detection in Source Code Using Deep Representation Learning”, 17th IEEE International Conference on Machine Learning and Applications, (2018), 7 pgs. [cited by applicant]
Sanchez-Lengeling, Benjamin, “Evaluating Attribution for Graph Neural Networks”, 34th Conference on Neural Information Processing Systems (NeurIPS 2020), Vancouver, Canada, (2020), 13 pgs. [cited by applicant]
Selvaraju, Ramprassaath R., “Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization”, Proceedings of the IEEE International Conference on Computer Vision (ICCV), (2017), 618-626. [cited by applicant]
Smilkov, Daniel, “SmoothGrad: removing noise by adding noise”, Sharper sensitivity maps: removing noise by adding noise, (Jun. 12, 2017), 10 pgs. [cited by applicant]
Warnecke, Alexander, “Evaluating Explanation Methods for Deep Learning in Security”, IEEE European Symposium on Security and Privacy (EuroSandP), (2020), 17 pgs. [cited by applicant]
Wu, Zonghan, “A Comprehensive Survey on Graph Neural Networks”, Journal of Latex Class Files, vol. XX, No. XX, (Dec. 4, 2019), 22 pgs. [cited by applicant]
Yamaguchi, Fabian, “Modeling and Discovering Vulnerabilities with Code Property Graphs”, IEEE Symposium on Security and Privacy, [Online]. Retrieved from the Internet: URL: https: comsecuris.com papers 06956589.pdf, (20… [cited by applicant]
Ying, Rex, “GNNExplainer: Generating Explanations for Graph Neural Networks”, 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada., (2019), 12 pgs. [cited by applicant]
Zhou, Yaqin, “Devign: Effective Vulnerability Identification by Learning Comprehensive Program Semantics via Graph Neural Networks”, 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Ca… [cited by applicant]
Zou, Deqing, “Interpreting Deep Learning-based Vulnerability Detector Predictions Based on Heuristic Searching”, ACM Trans. Softw. Eng. Methodol., vol. 37, No. 4, Article 111, (Aug. 2018), 33 pgs. [cited by applicant]