IP Library › Granted Patent US 12,547,674

Granted Patent B2

US 12,547,674 · App. 17/363,043 · Granted Feb 10, 2026

Adversarial image generator

Inventors: Quanfu Fan (Lexington, MA); Sijia Liu (Somerville, MA); Gaoyuan Zhang (Medford, MA); Kaidi Xu (Boston, MA)

Assignee: International Business Machines Corporation

G06F18/2148 G06F3/1208 G06F3/1226 G06N20/00

View Patent ↗

Loading inventors, assignments & file history…

Monitor This Case

Get email alerts when status or documents change.

Order Certified Copies

Most orders are placed with the USPTO same day — all within 24 business hours.

Order via The Patent Place →

Pre-filled with this patent's details

Quick Facts

Patent No.: US 12,547,674
App. No.: 17/363,043
Granted: Feb 10, 2026
Kind: B2

External Links

USPTO Patent Center↗ Full Patent PDF↗

Abstract

Adversarial patches can be inserted into sample pictures by an adversarial image generator to realistically depict adversarial images. The adversarial image generator can be utilized to train an adversarial patch generator by inserting generated patches into sample pictures, and submitting the resulting adversarial images to object detection models. This way, the adversarial patch generator can be trained to generate patches capable of defeating object detection models.

Claims (87)

1 . A method, comprising:

training an adversarial image generator;

receiving a sample picture, the sample picture depicting an object;

receiving an adversarial patch; and

generating, via the adversarial image generator, an adversarial image, wherein:

the adversarial image depicts the adversarial patch in the sample picture; and

the generating includes:

transforming the adversarial patch using environmental attributes of the sample picture, resulting in a transformed adversarial patch that is more similar than the adversarial patch to the sample picture; and

inserting the transformed adversarial patch into the picture, resulting in the adversarial image, wherein the training the adversarial image generator includes:

receiving a raw training patch;

receiving a first training picture, the first training picture including an embedded training patch; and

mapping a first transformation between the raw training patch and the embedded training patch.

2 . The method of claim 1 , wherein the training the adversarial image generator further includes:

identifying a first set of environmental attributes, the first set of environmental attributes associated with the depicted training patch of the first training picture;

receiving a second training picture, the second training picture depicting the training patch;

mapping a second transformation between the received training patch and the depicted training patch of the second training picture;

identifying a second set of environmental attributes, the second set of environmental attributes associated with the depicted training patch of the second training picture; and

updating a model to output the first transformation based on receipt of the first set of environmental attributes and to output the second transformation based on receipt of the second set of environmental attributes.

3 . The method of claim 2 , further comprising:

receiving a third set of environmental attributes, the third set of environmental attributes associated with the sample patch depicted in the sample picture;

inputting the third set of environmental attributes into the model; and

receiving an output transformation from the model based on the inputting, wherein the transforming the adversarial patch is based on the output transformation.

4 . The method of claim 2 , wherein the first set of environmental attributes describes lighting conditions of the first training picture.

5 . The method of claim 2 , wherein the first set of environmental attributes describes a printer color discrepancy.

6 . The method of claim 1 , further comprising training an adversarial patch generator, the training the adversarial patch generator including updating a weight by an amount, wherein the receiving the adversarial patch includes generating the adversarial patch via the adversarial patch generator.

7 . The method of claim 1 , further comprising training an object detection model to detect the object, the training the object detection model including:

submitting the adversarial image to the object detection model;

receiving an output from the object detection model as a result of the submitting the adversarial image;

evaluating a performance of the object detection model based on the output; and

updating the object detection model based on the performance.

8 . A system, comprising:

a memory; and

a processor coupled to the memory, the processor configured to:

train an adversarial image generator;

receive a sample picture, the sample picture depicting an object;

receive an adversarial patch; and

generate, via the adversarial image generator, an adversarial image, wherein:

the adversarial image depicts the adversarial patch in the sample picture; and the generating includes:

inserting the transformed adversarial patch into the picture, resulting in the adversarial image, wherein the training the adversarial image generator includes:

receiving a raw training patch;

receiving a first training picture, the first training picture including an embedded training patch; and

mapping a first transformation between the raw training patch and the embedded training patch.

9 . The system of claim 8 , wherein the training the adversarial image generator further includes:

identifying a first set of environmental attributes, the first set of environmental attributes associated with the depicted training patch of the first training picture;

receiving a second training picture, the second training picture depicting the training patch;

mapping a second transformation between the received training patch and the depicted training patch of the second training picture;

identifying a second set of environmental attributes, the second set of environmental attributes associated with the depicted training patch of the second training picture; and

10 . The system of claim 9 , wherein the processor is further configured to:

receive a third set of environmental attributes, the third set of environmental attributes associated with the sample patch depicted in the sample picture;

input the third set of environmental attributes into the model; and

receive an output transformation from the model based on the inputting, wherein the transforming the adversarial patch is based on the output transformation.

11 . The system of claim 8 , wherein the processor is further configured to train an adversarial patch generator, the training the adversarial patch generator including updating a weight by an amount, wherein the receiving the adversarial patch includes generating the adversarial patch via the adversarial patch generator.

12 . The system of claim 8 , wherein the processor is further configured to train an object detection model to detect the object, the training the object detection model including:

submitting the adversarial image to the object detection model;

receiving an output from the object detection model as a result of the submitting the adversarial image;

evaluating a performance of the object detection model based on the output; and

updating the object detection model based on the performance.

13 . A computer program product, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to:

train an adversarial image generator;

receive a sample picture, the sample picture depicting an object;

receive an adversarial patch; and

generate, via the adversarial image generator, an adversarial image, wherein:

the adversarial image depicts the adversarial patch in the sample picture; and

the generating includes:

inserting the transformed adversarial patch into the picture, resulting in the adversarial image, wherein the training the adversarial image generator includes:

receiving a raw training patch;

receiving a first training picture, the first training picture including an embedded training patch; and

mapping a first transformation between the raw training patch and the embedded training patch.

14 . The computer program product of claim 13 , wherein the training the adversarial image generator further includes:

identifying a first set of environmental attributes, the first set of environmental attributes associated with the depicted training patch of the first training picture;

receiving a second training picture, the second training picture depicting the training patch;

mapping a second transformation between the received training patch and the depicted training patch of the second training picture;

identifying a second set of environmental attributes, the second set of environmental attributes associated with the depicted training patch of the second training picture; and

15 . The computer program product of claim 14 , wherein the instructions further cause the computer to:

receive a third set of environmental attributes, the third set of environmental attributes associated with the sample patch depicted in the sample picture;

input the third set of environmental attributes into the model; and

receive an output transformation from the model based on the inputting, wherein the transforming the adversarial patch is based on the output transformation.

16 . The computer program product of claim 13 , wherein the instructions further cause the computer to train an adversarial patch generator, the training the adversarial patch generator including updating a weight by an amount, wherein the receiving the adversarial patch includes generating the adversarial patch via the adversarial patch generator.

17 . The computer program product of claim 13 , wherein the instructions further cause the computer to train an object detection model to detect the object, the training the object detection model including:

submitting the adversarial image to the object detection model;

receiving an output from the object detection model as a result of the submitting the adversarial image;

evaluating a performance of the object detection model based on the output; and

updating the object detection model based on the performance.

Assignments (1)

ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 30, 2021

From: FAN, QUANFU; LIU, SIJIA; ZHANG, GAOYUAN; XU, KAIDI

To: INTERNATIONAL BUSINESS MACHINES CORPORATION

Reel/Frame 056713/0157 →

Continuity (1)

Related Publication 20230004754A1 · Jan 5, 2023

References Cited (33)

US 10825148B2 · Tagra et al. · 2020 [cited by applicant]

US 10839268B1 · Ardulov et al. · 2020 [cited by applicant]

US 10910099B2 · Xu et al. · 2021 [cited by applicant]

US 11151703B2 · Sargent · 2021 [cited by applicant]

US 20210103814A1 · Tsiligkaridis · 2021 [cited by examiner]

US 20230206601A1 · Metzen · 2023 [cited by examiner]

CN 111340008A · 2020 [cited by applicant]

DE 102020004960A1 · 2021 [cited by applicant]

Eykholt et al., “Robust Physical-World Attacks on Deep Learning Visual Classification,” In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 1625-1634. [cited by applicant]

Sharif et al., “Accessorize to a Crime: Real and Stealthy Attacks on State-of-the-Art Face Recognition,” ACM, In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, 2016, pp. 1528-1540. [cited by applicant]

Xu et al., “Adversarial T-shirt! Evading Person Detectors in A Physical World,” arXiv:1910.11099v3 [cs.CV], Jul. 7, 2020, 23 pages, https://arxiv.org/pdf/1910.11099.pdf. [cited by applicant]

Pautov et al., “On adversarial patches: real-world attack on ArcFace-100 face recognition system,” 2019 International Multi-Conference on Engineering, Computer and Information Sciences (SIBIRCON), arXiv:1910.07067v3 [cs… [cited by applicant]

“AI-Aware Engine for Matching Learners with Mentors,” IP.Com., An IP.com Prior Art Database Technical Disclosure, IP.com No. IPCOM000261341D, IP.com Electronic Publication Date: Feb. 24, 2020, 5 pages. [cited by applicant]

“A Method for social and location-aware group sequential recommendation,” IP.Com, An IP.com Prior Art Database Technical Disclosure, IP.com No. IPCOM000252662D, IP.com Electronic Publication Date: Feb. 1, 2018, 4 pages. [cited by applicant]

“A Method to Manage Patches on Multiple Computer Systems,” IP.Com, An IP.com Prior Art Database Technical Disclosure, IP.com No. IPCOM000259485D, IP.com Electronic Publication Date: Aug. 15, 2019, 3 pages. [cited by applicant]

McCoyd et al., “Minority Reports Defense: Defending Against Adversarial Patches,” arXiv:2004.13799v1 [cs.LG] Apr. 28, 2020, Printed May 14, 2021, 9 pages. [cited by applicant]

Yang et al., “Design and Interpretation of Universal Adversarial Patches in Face Detection,” arXiv:1912.05021v3 [cs.CV] Jul. 17, 2020, Printed May 14, 2021, 20 pages. [cited by applicant]

Metzen et al., “Efficient Certified Defenses Against Patch Attacks on Image Classifiers,” arXiv:2102.04154v1 [cs.LG] Feb. 8, 2021, Published as a conference paper at ICLR 2021, 17 pages. [cited by applicant]

Brown et al., “Adversarial Patch,” 31st Conference on Neural Information Processing Systems (NIPS 2017), 2017, 5 pages. [cited by applicant]

Thys et al., “Fooling automated surveillance cameras: adversarial patches to attack person detection,” CVPR Workshops, arXiv:1904.08653v1 [cs.CV], Apr. 18, 2019, 7 pages. [cited by applicant]

Zhao et al., “Seeing isn't Believing: Towards More Robust Adversarial Attack Against Real World Object Detectors,” CSS19, arXiv:1812.10217v3 [cs.CV], Sep. 4, 2019, 16 pages. [cited by applicant]

Zhang et al., “Deep Mutual Learning,” CVPR 2018, arXiv:1706.00384v1 [cs.CV], Jun. 1, 2017, 10 pages. [cited by applicant]

Lan et al., “Knowledge Distillation by On-the-Fly Native Ensemble,” NIPS 2018, arXiv:1806.04606v2 [cs.CV], Sep. 8, 2018, 11 pages. [cited by applicant]

Chen et al., “Can 3D Adversarial Logos Cloak Humans?,” arXiv:2006.14655v1 [cs.LG], Jun. 25, 2020, 10 pages. [cited by applicant]

Bor-Chun Chen et al., “Toward Realistic Image Compositing with Adversarial Learning,” Printed Jun. 29, 2021, 10 pages. [cited by applicant]

Athalye et al., “Synthesizing Robust Adversarial Examples,” arXiv:1707.07397v3 [cs.CV], Jun. 7, 2018, 19 pages. [cited by applicant]

Wu et al., “Making an Invisibility Cloak: Real World Adversarial Attacks on Object Detectors,” arXiv:1910.14667v2 [cs.CV], Jul. 22, 2020, 16 pages. [cited by applicant]

“Generating Realistic Physical Adversarial Examples by Patch Transformer Networks,” CVPR 2021 Submission #9922, 2021, 13 pages. [cited by applicant]

Athalye et al., “Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples,” arXiv:1802.00420v4 [cs.LG], Jul. 31, 2018, 12 pages. [cited by applicant]

Carlini et al., “Audio Adversarial Examples: Targeted Attacks on Speech-to-Text,” IEEE, In 2018 IEEE Security and Privacy Workshops (SPW), 2018, pp. 1-7. [cited by applicant]

Xu et al., “Structured Adversarial Attack: Towards General Implementation and Better Interpretability,” In International Conference on Learning Representations, 2019, 21 pages. [cited by applicant]

List of IBM Patents or Applications Treated as Related, Dated Jun. 29, 2021, 2 pages. [cited by applicant]

Fan et al., “Detecting Hybdrid-Distance Adversarial Patches,” U.S. Appl. No. 17/363,054, filed Jun. 30, 2021. [cited by applicant]