IP Library › Granted Patent US 12,242,613
Granted Patent B2
US 12,242,613 · App. 16/948,762 · Granted Mar 4, 2025

Automated evaluation of machine learning models

Inventors: Mathieu Sinn (Mulhuddart, IE); Beat Buesser (Mulhuddart, IE); Ngoc Minh Tran (Dublin, IE); Killian Levacher (Dublin, IE); Hessel Tuinhof (Dublin, IE)
G06F21/577G06N20/00G06F2221/034
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,242,613
App. No.
16/948,762
Granted
Mar 4, 2025
Kind
B2
Abstract

Various embodiments are provided for automated evaluation of machine learning models in a computing environment by one or more processors in a computing system. A level of robustness of a machine learning model against adversarial whitebox operations may be evaluated and determined by applying a data set used for testing the machine learning model, one or more adversarial operation objectives, an adversarial threat model, and a selected number of hyperparameters. Results from the adversarial operation may be analyzed and a modified machine learning model may be generated while performing the evaluating and determining.

Claims (102)

1. A method for automated evaluation of machine learning models in a computing environment by one or more processors comprising:

receiving a machine learning model, a data set used for testing the machine learning model, one or more adversarial operation objectives, an adversarial threat model, and a selected number of hyperparameters;

automatically evaluating and determining a level of robustness of the machine learning model against an adversarial whitebox operation and analyzing results from the adversarial whitebox operation using the data set, the one or more adversarial operation objectives, the adversarial threat model, and the selected number of hyperparameters, wherein the analyzing includes:

localizing a plurality of masked gradients within a graphical representation of the machine learning model, wherein the plurality of masked gradients is localized using brute-force or AI search on one or more nodes causing gradient masking;

unmasking one or more of the plurality of masked gradients by applying backward-pass differentiable approximations or reparameterizations;

diagnosing gradient information of one or more unmasked gradients and a current adversarial operation loss function;

identifying one or more parameter optimizations based on the diagnosing of the gradient information;

determining a first robustness score for the machine learning model by which the machine learning model is ranked, wherein the first robustness score indicates a level of security of the machine learning model against the adversarial whitebox operation;

generating an evaluation summary, wherein the evaluation summary includes the first robustness score, a provenance of each adversarial sample, and any issues that are recommended to be resolved manually;

generating a modified machine learning model by modifying an unmasking of gradients of the machine learning model by applying the backward-pass differentiable approximations or the reparameterizations by adjusting the one or more parameter optimizations previously identified;

determining a second robustness score for the modified machine learning model by evaluating the level of robustness of the modified machine learning model against the adversarial whitebox operations using partial inputs from a previous evaluation of the machine learning model;

ranking the modified machine learning model according to the level of robustness using the second robustness score, wherein a machine learning operation is initialized to learn and store the level of robustness of the machine learning model against the adversarial whitebox operations based on the machine learning model, the data set used for testing the machine learning model, the one or more adversarial objectives, the adversarial threat model, and a selected number of hyperparameters.

2. The method of claim 1 , wherein the diagnosing the gradient information for the plurality of masked gradients includes identifying at least a type of gradient masking for each of the plurality of masked gradients within the graphical representation of the machine learning model, wherein the type of gradient masking includes at least one or more of, shattered gradients, stochastic gradients, or vanishing gradients.

3. The method of claim 1 , further comprising:

generating one or more log messages, wherein the one or more log messages includes details with respect to the localization and the diagnosis of the plurality of masked gradients, operations performed to unmask the plurality of masked gradients, and a list of any components for which the unmasking is unsuccessful.

4. The method of claim 1 , further comprising:

invoking a loss function composer in response to inconsistency or poor convergence in the current adversarial operation loss function, wherein the loss function composer utilizes output of the machine learning model and the one or more unmasked gradients to compose loss functions consistent with the one or more adversarial operation objectives.

5. The method of claim 4 , further comprising:

executing the one or more adversarial operation objectives, the adversarial threat model, and the selected number of hyperparameters on the machine learning model;

applying one or more defenses in response to the executing the one or more adversarial operation objectives, the adversarial threat model, and the selected number of hyperparameters on the machine learning model;

adjusting the one or more adversarial operation objectives, the adversarial threat model, a selected number of hyperparameters, and the data set used for testing the machine learning model; and

modifying the unmasking gradients of the machine learning model, a loss function, and the one or more adversarial operation objections, wherein the loss function is modified based on the loss function composes in response to the inconsistency or the poor convergence in the current adversarial operation loss function.

6. The method of claim 1 , further comprising:

invoking a meta optimizer in response to the diagnosing of the gradient information, wherein the meta optimizer utilizes different search strategies or different optimizers to tune the hyperparameters targeting a loss function towards optimal performance with respect to the one or more adversarial operation objectives.

7. The method of claim 1 , wherein the modified machine learning model is generated while in a process of evaluations by changing a compute graph in order to further unmask gradients.

8. The method of claim 1 , wherein the machine learning model is modified to prevent gradient masking such that the level of robustness is not overestimated.

9. The method of claim 1 , wherein the initializing of the machine learning operation further comprises:

collecting feedback in relation to automatically evaluating and determining of the level of robustness of the machine learning model against the adversarial whitebox operations to generate the modified machine learning model while performing at least a portion of the evaluating and determining.

10. A system for automated evaluation of machine learning models in a computing environment, comprising:

one or more computers with executable instructions that when executed cause the system to:

receiving a machine learning model, a data set used for testing the machine learning model, one or more adversarial operation objectives, an adversarial threat model, and a selected number of hyperparameters;

automatically evaluating and determining a level of robustness of the machine learning model against an adversarial whitebox operation and analyzing results from the adversarial whitebox operation using the data set, the one or more adversarial operation objectives, the adversarial threat model, and the selected number of hyperparameters, wherein the analyzing includes:

localizing a plurality of masked gradients within a graphical representation of the machine learning model, wherein the plurality of masked gradients is localized using brute-force or AI search on one or more nodes causing gradient masking;

unmasking one or more of the plurality of masked gradients by applying backward-pass differentiable approximations or reparameterizations;

diagnosing gradient information of one or more unmasked gradients and a current adversarial operation loss function;

identifying one or more parameter optimizations based on the diagnosing of the gradient information;

determining a first robustness score for the machine learning model by which the machine learning model is ranked, wherein the first robustness score indicates a level of security of the machine learning model against the adversarial whitebox operation;

generating an evaluation summary, wherein the evaluation summary includes the first robustness score, a provenance of each adversarial sample, and any issues that are recommended to be resolved manually;

generating a modified machine learning model by modifying an unmasking of gradients of the machine learning model by applying the backward-pass differentiable approximations or the reparameterizations by adjusting the one or more parameter optimizations previously identified;

determining a second robustness score for the modified machine learning model by evaluating the level of robustness of the modified machine learning model against the adversarial whitebox operations using partial inputs from a previous evaluation of the machine learning model;

ranking the modified machine learning model according to the level of robustness using the second robustness score, wherein a machine learning operation is initialized to learn and store the level of robustness of the machine learning model against the adversarial whitebox operations based on the machine learning model, the data set used for testing the machine learning model, the one or more adversarial objectives, the adversarial threat model, and a selected number of hyperparameters.

11. The system of claim 10 , wherein the executable instructions when executed cause the system to diagnose the gradient information for the plurality of masked gradients includes identifying at least a type of gradient masking for each of the plurality of masked gradients within the graphical representation of the machine learning model, wherein the type of gradient masking includes at least one or more of, shattered gradients, stochastic gradients, or vanishing gradients.

12. The system of claim 10 , wherein the executable instructions when executed cause the system to:

generate one or more log messages, wherein the one or more log messages includes details with respect to the localization and the diagnosis of the plurality of masked gradients, operations performed to unmask the plurality of masked gradients, and a list of any components for which the unmasking is unsuccessful.

13. The system of claim 10 , wherein the executable instructions when executed cause the system to:

invoke a loss function composer in response to inconsistency or poor convergence in the current adversarial operation loss function, wherein the loss function composer utilizes output of the machine learning model and the one or more unmasked gradients to compose loss functions consistent with the one or more adversarial operation objectives.

14. The system of claim 13 , wherein the executable instructions when executed cause the system to:

execute the one or more adversarial operation objectives, the adversarial threat model, and the selected number of hyperparameters on the machine learning model;

apply one or more defenses in response to the executing the one or more adversarial operation objectives, the adversarial threat model, and the selected number of hyperparameters on the machine learning model;

adjust the one or more adversarial operation objectives, the adversarial threat model, a selected number of hyperparameters, and the data set used for testing the machine learning model; and

modify the unmasking gradients of the machine learning model, a loss function, and the one or more adversarial operation objections, wherein the loss function is modified based on the loss function composes in response to the inconsistency or the poor convergence in the current adversarial operation loss function.

15. The system of claim 10 , wherein the executable instructions when executed cause the system to:

invoke a meta optimizer in response to the diagnosing of the gradient information, wherein the meta optimizer utilizes different search strategies or different optimizers to tune the hyperparameters targeting a loss function towards optimal performance with respect to the one or more adversarial operation objectives.

16. A computer program product for automated evaluation of machine learning models in a computing environment, the computer program product comprising:

one or more computer readable storage media, and program instructions collectively stored on the one or more computer readable storage media, the program instruction comprising:

program instructions to receive a machine learning model, a data set used for testing the machine learning model, one or more adversarial operation objectives, an adversarial threat model, and a selected number of hyperparameters;

program instructions to automatically evaluating and determining a level of robustness of the machine learning model against an adversarial whitebox operation and analyzing results from the adversarial whitebox operation using the data set, the one or more adversarial operation objectives, the adversarial threat model, and the selected number of hyperparameters, wherein the analyzing includes:

localizing a plurality of masked gradients within a graphical representation of the machine learning model, wherein the plurality of masked gradients is localized using brute-force or AI search on one or more nodes causing gradient masking;

unmasking one or more of the plurality of masked gradients by applying backward-pass differentiable approximations or reparameterizations;

diagnosing gradient information of one or more unmasked gradients and a current adversarial operation loss function;

identifying one or more parameter optimizations based on the diagnosing of the gradient information;

determining a first robustness score for the machine learning model by which the machine learning model is ranked, wherein the first robustness score indicates a level of security of the machine learning model against the adversarial whitebox operation;

generating an evaluation summary, wherein the evaluation summary includes the first robustness score, a provenance of each adversarial sample, and any issues that are recommended to be resolved manually;

generating a modified machine learning model by modifying an unmasking of gradients of the machine learning model by applying the backward-pass differentiable approximations or the reparameterizations by adjusting the one or more parameter optimizations previously identified;

determining a second robustness score for the modified machine learning model by evaluating the level of robustness of the modified machine learning model against the adversarial whitebox operations using partial inputs from a previous evaluation of the machine learning model;

ranking the modified machine learning model according to the level of robustness using the second robustness score, wherein a machine learning operation is initialized to learn and store the level of robustness of the machine learning model against the adversarial whitebox operations based on the machine learning model, the data set used for testing the machine learning model, the one or more adversarial objectives, the adversarial threat model, and a selected number of hyperparameters.

17. The computer program product of claim 16 , further including program instructions to diagnose the gradient information for the plurality of masked gradients includes identifying at least a type of gradient masking for each of the plurality of masked gradients within the graphical representation of the machine learning model, wherein the type of gradient masking includes at least one or more of, shattered gradients, stochastic gradients, or vanishing gradients.

18. The computer program product of claim 16 , further including program instructions to:

generate one or more log messages, wherein the one or more log messages includes details with respect to the localization and the diagnosis of the plurality of masked gradients, operations performed to unmask the plurality of masked gradients, and a list of any components for which the unmasking is unsuccessful.

19. The computer program product of claim 16 , further including program instructions to:

invoke a loss function composer in response to inconsistency or poor convergence in the current adversarial operation loss function, wherein the loss function composer utilizes output of the machine learning model and the one or more unmasked gradients to compose loss functions consistent with the one or more adversarial operation objectives.

20. The computer program product of claim 19 , further including program instructions to:

execute the one or more adversarial operation objectives, the adversarial threat model, and the selected number of hyperparameters on the machine learning model;

apply one or more defenses in response to the executing the one or more adversarial operation objectives, the adversarial threat model, and the selected number of hyperparameters on the machine learning model;

adjust the one or more adversarial operation objectives, the adversarial threat model, a selected number of hyperparameters, and the data set used for testing the machine learning model; and

modify the unmasking gradients of the machine learning model, a loss function, and the one or more adversarial operation objections, wherein the loss function is modified based on the loss function composes in response to the inconsistency or the poor convergence in the current adversarial operation loss function.

21. The computer program product of claim 16 , further including program instructions to:

invoke a meta optimizer in response to the diagnosing of the gradient information, wherein the meta optimizer utilizes different search strategies or different optimizers to tune the hyperparameters targeting a loss function towards optimal performance with respect to the one or more adversarial operation objectives.

22. A method for automated evaluation of machine learning models in a computing environment by one or more processors comprising:

receiving a machine learning model, a data set used for testing the machine learning model, one or more adversarial operation objectives, an adversarial threat model, and a selected number of hyperparameters;

automatically evaluating and determining a level of robustness of the machine learning model against an adversarial whitebox operation and analyzing results from the adversarial whitebox operation using the data set, the one or more adversarial operation objectives, the adversarial threat model, and the selected number of hyperparameters, wherein the analyzing includes:

localizing a plurality of masked gradients within a graphical representation of the machine learning model;

unmasking one or more of the plurality of masked gradients by applying backward-pass differentiable approximations or reparameterizations;

diagnosing gradient information of one or more unmasked gradients and a current adversarial operation loss function;

identifying one or more parameter optimizations based on the diagnosing of the gradient information;

determining a first robustness score for the machine learning model by which the machine learning model is ranked, wherein the first robustness score indicates a level of security of the machine learning model against the adversarial whitebox operation; and

generating a modified machine learning model by modifying an unmasking of gradients of the machine learning model by applying the backward-pass differentiable approximations or the reparameterizations by adjusting the one or more parameter optimizations previously identified.

23. The method of claim 22 , further comprising:

generating an evaluation summary, wherein the evaluation summary includes the first robustness score, a provenance of each adversarial sample, and any issues that are recommended to be resolved manually.

24. The method of claim 22 , further comprising:

determining a second robustness score for the modified machine learning model by evaluating the level of robustness of the modified machine learning model against the adversarial whitebox operations using partial inputs from a previous evaluation of the machine learning model; and

ranking the modified machine learning model according to the level of robustness using the second robustness score, wherein a machine learning operation is initialized to learn and store the level of robustness of the machine learning model against the adversarial whitebox operations based on the machine learning model, the data set used for testing the machine learning model, the one or more adversarial objectives, the adversarial threat model, and a selected number of hyperparameters.

25. A system for automated evaluation of machine learning models in a computing environment, comprising:

one or more computers with executable instructions that when executed cause the system to:

receiving a machine learning model, a data set used for testing the machine learning model, one or more adversarial operation objectives, an adversarial threat model, and a selected number of hyperparameters;

automatically evaluating and determining a level of robustness of the machine learning model against an adversarial whitebox operation and analyzing results from the adversarial whitebox operation using the data set, the one or more adversarial operation objectives, the adversarial threat model, and the selected number of hyperparameters, wherein the analyzing includes:

localizing a plurality of masked gradients within a graphical representation of the machine learning model;

unmasking one or more of the plurality of masked gradients by applying backward-pass differentiable approximations or reparameterizations;

diagnosing gradient information of one or more unmasked gradients and a current adversarial operation loss function;

identifying one or more parameter optimizations based on the diagnosing of the gradient information;

determining a first robustness score for the machine learning model by which the machine learning model is ranked, wherein the first robustness score indicates a level of security of the machine learning model against the adversarial whitebox operation; and

generating a modified machine learning model by modifying an unmasking of gradients of the machine learning model by applying the backward-pass differentiable approximations or the reparameterizations by adjusting the one or more parameter optimizations previously identified.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 4, 2024
From: SINN, MATHIEU; BUESSER, BEAT; TRAN, NGOC MINH; LEVACHER, KILLIAN; TUINHOF, HESSEL
To: INTERNATIONAL BUSINESS MACHINES CORPORATION
Reel/Frame 069487/0470 →
Continuity (1)
Related Publication 20220100867A1 · Mar 31, 2022
References Cited (54)
US 10657259B2 · Lee et al. · 2020 [cited by applicant]
US 10896664B1 · Buesser · 2021 [cited by examiner]
US 11036857B2 · Tran · 2021 [cited by examiner]
US 11200456B2 · Pinkovich · 2021 [cited by examiner]
US 11288408B2 · Buesser · 2022 [cited by examiner]
US 11334671B2 · Buesser · 2022 [cited by examiner]
US 11494496B2 · Grosse · 2022 [cited by examiner]
US 11636284B2 · Ghosh · 2023 [cited by examiner]
US 11783230B2 · Brower · 2023 [cited by examiner]
US 11836256B2 · Chen · 2023 [cited by examiner]
US 20180046915A1 · Sun · 2018 [cited by examiner]
US 20190080089A1 · Chen · 2019 [cited by applicant]
US 20190188562A1 · Edwards et al. · 2019 [cited by applicant]
US 20200143045A1 · Joye · 2020 [cited by examiner]
US 20200151505A1 · Saito et al. · 2020 [cited by applicant]
US 20200159924A1 · Tran · 2020 [cited by examiner]
US 20200167471A1 · Rouhani et al. · 2020 [cited by applicant]
US 20200201993A1 · Gonnet et al. · 2020 [cited by applicant]
US 20200286095A1 · Anunciacao · 2020 [cited by examiner]
US 20200380300A1 · Zhang · 2020 [cited by examiner]
US 20200387836A1 · Nasr-Azadani · 2020 [cited by examiner]
US 20200410228A1 · Wang · 2020 [cited by examiner]
US 20210048806A1 · Fainekos · 2021 [cited by examiner]
CN 108681250A · 2018 [cited by applicant]
EP 3611854A1 · 2020 [cited by applicant]
WO 2020028036A1 · 2020 [cited by applicant]
Title: Adversarial Robustness Toolbox v1.0.0 Author(s): Maria-Irina Nicolae, Mathieu Sinn, Minh Ngoc Tran, Beat Buesser, Ambrish Rawat, Martin Wistuba, Year: 2019 Publisher: arxiv.org. [cited by examiner]
CWE Content Team, CWE-1039 “Automated Recognition Mechanism with Inadequate Detection or Handling of Adversarial Input Perturbations”; Mar. 12, 2018. https://cwe.mitre.org/data/definitions/1039.html. [cited by applicant]
R. Shankar et al., “Adversarial Machine Learning—Industry perspectives”, arXiv:2002.05646, 2020. May 2020. (7 pages). [cited by applicant]
Security in Development—The IBM Secure Engineering Framework. 2018 (28 pages) https://www.redbooks.ibm.com/redpapers/pdfs/redp4641.pdf. [cited by applicant]
“White Paper on Artificial Intelligence A European Approach to Excellence and Trust” Feb. 19, 2020; https://ec.europa.eu/info/sites/info/files/commission-white-paper-artificial-intelligence-feb2020_en.pdf (26 pages). [cited by applicant]
W. He et al., “Adversarial example defenses: Ensembles of weak defenses are not strong”, arXiv:1706.04701, 2017. (11 pages). [cited by applicant]
N. Carlini, D. Wagner, “Adversarial examples are not easily detected: Bypassing ten detection methods”, AlSec Workshop, 2017. (14 pages). [cited by applicant]
A. Athalye et al., “Obfuscated gradients give a false sense of security: circumventing defenses to adversarial examples”, ICML, 2018. arXiv:1802.00420v4 2018. (12 pages). [cited by applicant]
N. Carlini et al., “On evaluating adversarial robustness”, arXiv:1902.06705, 2019. (24 pages). [cited by applicant]
F. Tramer et al., “On adaptive attacks to adversarial example defences”, arXiv:2002.08347, 2020. (43 pages). [cited by applicant]
N. Papernot et al., “Technical Report on the CleverHans v2.1.0 Adversarial Examples Library”, 2018. [cited by applicant]
Nicolae et al., “Adversarial Robustness Toolbox v1.2.0”, https://github.com/ibm/adversarial-robustness-toolbox, 2018. [cited by applicant]
J. Rauber et al., “Foolbox Native: Fast adversarial attacks to benchmark the robustness of machine learning models in PyTorch, TensorFlow, and JAX”, https://github.com/bethgelab/foolbox, 2020. [cited by applicant]
B. Biggio et al., “Evasion attacks against machine learning at test time”, ECML-PKDD, 2013. (16 pages). [cited by applicant]
C. Szegedy et al., “Intriguing properties of neural networks”, arXiv:1312.6199, 2013. (10 pages). [cited by applicant]
M. Nicolae et al., “Adversarial Robustness Toolbox v1.0.0 ”, arXiv:1807.01069, 2018. (34 pages). [cited by applicant]
A. Madry et al., “Towards deep learning models resistant to adversarial attacks”, ICLR, 2018. (28 pages). [cited by applicant]
J. Cohen et al., “Certified Adversarial Robustness via Randomized Smoothing”, arXiv:1902.02918, 2019. (36 pages). [cited by applicant]
P. Chen et al., “EAD: Elastic-net attacks to deep neural networks via adversarial examples”, AAAI, 2018. (19 pages). [cited by applicant]
N. Carlini, D. Wagner, “Towards evaluating the robustness of neural networks”, IEEE S&P 2017. (19 pages). [cited by applicant]
P. Chen et al., “ZOO: Zeroth Order Optimization based black-box attacks to deep neural network without training substitute models”, AlSec workshop, 2017. (12 pages). [cited by applicant]
J. Chen et al., “HopSkipJumpAttack: a query-efficient decision-based attack”, arXiv:1904.02144, 2019. [cited by applicant]
N Papernot et al., “Practical black-box attacks against machine learning”, Asia CCS, 2017. (14 pages). [cited by applicant]
F. Tramer et al., “Ensemble adversarial training: Attacks and defenses”, ICLR, 2018. (22 pages). [cited by applicant]
I. Goodfellow, “New CleverHans feature: Better adversarial robustness evaluations with attack bundling”, arXiv:1811.03685, 2018. (6 pages). [cited by applicant]
M. Sharif et al., “Accessorize to a crime: Real and stealthy attacks on state-of-the-art face recognition”, CCS, 2016. (13 pages). [cited by applicant]
A. Athalye et al., “Synthesizing robust adversarial examples”, arXiv:1707.07397, 2017 (20 pages). [cited by applicant]
Y. Bengio et al., “Estimating or propagating gradients through stochastic neurons for conditional computation”, arXiv:1308.3432, 2013. (12 pages). [cited by applicant]
Cited By (1)
US 12,694,331