IP Library › Granted Patent US 12,657,928
Granted Patent B2
US 12,657,928 · App. 18/172,823 · Granted Jun 16, 2026

Approximately-paired simulation-to-real image translation

Inventors: Ashish Shrivastava (San Jose, CA); Charles Yingjia Zhang (Ottawa, CA)
Assignee: GM CRUISE HOLDINGS LLC
G06V20/58B60W60/001B60W2420/403
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,657,928
App. No.
18/172,823
Granted
Jun 16, 2026
Kind
B2
Abstract

Disclosed are embodiments for facilitating approximately-paired simulation-to-real image translation. In some aspects, a method includes receiving, by a processing device performing training on a model, an approximately-paired image pair comprising a real image and a simulated image, wherein the simulated image is generated from the real image using contextual data of the real image; determining, by the processing device using a style encoder of the model, a style difference between a first style vector of the simulated image and a second style vector of the real image, wherein the first style vector and the second style vector encode style features of the simulated image and the real image using a style encoder; and inputting the style difference and the simulated image to a generative adversarial network (GAN) of the model to train the GAN to generate a post-processed simulated image.

Claims (45)

1 . A method comprising:

receiving, by a processing device performing training on a model, an approximately-paired image pair comprising a real image and a simulated image, wherein the simulated image is generated from the real image using contextual data of the real image;

determining, by the processing device using a style encoder of the model, a style difference between a first style vector of the simulated image and a second style vector of the real image, wherein the first style vector and the second style vector encode style features of the simulated image and the real image using a style encoder; and

inputting the style difference and the simulated image to a generative adversarial network (GAN) of the model to train the GAN to generate a post-processed simulated image.

2 . The method of claim 1 , wherein the model comprises a machine learning model for an autonomous vehicle (AV).

3 . The method of claim 1 , wherein the style features comprise at least one of contrast, color balance, sharpness, image noise, lighting, or scene fidelity.

4 . The method of claim 1 , wherein the GAN comprises a generator neural network and a discriminator neural network, wherein the generator neural network is to translate the simulated image into the post-processed simulated image based on the style difference, and wherein the discriminator neural network is to distinguish between the post-processed simulated image and the real image.

5 . The method of claim 4 , wherein a trained version of the generator neural network is utilized during an inference phase to:

receive the simulated image and a style difference between the simulated image and a random real style image; and

generate the post-processed simulated image.

6 . The method of claim 1 , wherein the contextual data comprises metadata and label data of the real image, the metadata and the label data comprising one or more of identification of objects, a pose of a sensor, map location, environment conditions, and lighting.

7 . The method of claim 6 , wherein generating the simulated image from the real image further comprises:

utilizing the metadata and the label data corresponding to the real image to select assets corresponding to the objects in the real image, select a background corresponding to the real image, and select lighting corresponding to the real image;

render the simulated image using the selected assets, the selected background, and the selected lighting; and

indicate that the simulated image and the real image are the approximately-paired image pair.

8 . The method of claim 1 , wherein training the GAN comprises utilizing losses to optimize the GAN, the losses comprising at least one of an adversarial loss, a style classification loss, a style reconstruction loss, or a content loss.

9 . The method of claim 8 , wherein the content loss comprises a Noise-Contrastive Estimation (NCE) loss, an identity loss, and a luminance loss.

10 . An apparatus comprising:

one or more hardware processors to:

receive, by a processing device performing training on a model, an approximately-paired image pair comprising a real image and a simulated image, wherein the simulated image is generated from the real image using contextual data of the real image;

determine, by the processing device using a style encoder of the model, a style difference between a first style vector of the simulated image and a second style vector of the real image, wherein the first style vector and the second style vector encode style features of the simulated image and the real image using a style encoder; and

input the style difference and the simulated image to a generative adversarial network (GAN) of the model to train the GAN to generate a post-processed simulated image.

11 . The apparatus of claim 10 , wherein the style features comprise at least one of contrast, color balance, sharpness, image noise, lighting, or scene fidelity.

12 . The apparatus of claim 10 , wherein the GAN comprises a generator neural network and a discriminator neural network, wherein the generator neural network is to translate the simulated image into the post-processed simulated image based on the style difference, and wherein the discriminator neural network is to distinguish between the post-processed simulated image and the real image.

13 . The apparatus of claim 12 , wherein a trained version of the generator neural network is utilized during an inference phase to:

receive the simulated image and a style difference between the simulated image and a random real style image; and

generate the post-processed simulated image.

14 . The apparatus of claim 10 , wherein the contextual data comprises metadata and label data of the real image, the metadata and label data comprising one or more of identification of objects, a pose of a sensor, map location, environment conditions, and lighting.

15 . The apparatus of claim 14 , wherein generating the simulated image from the real image further comprises:

utilizing the metadata and the label data corresponding to the real image to select assets corresponding to the objects in the real image, select a background corresponding to the real image, and select lighting corresponding to the real image;

render the simulated image using the selected assets, the selected background, and the selected lighting; and

indicate that the simulated image and the real image are the approximately-paired image pair.

16 . A non-transitory computer-readable medium having stored thereon instructions that, when executed by one or more processors, cause the one or more processors to:

receive, by the one or more processors performing hosting a trained model, a simulated image and a real image, wherein the trained model is trained using approximately-paired image pairs comprising training simulated images generated from training real images using contextual data of the training real images;

extract, by the one or more processors using one or more style encoders of the trained model, a first style vector from the simulated image and a second style vector from the real image, wherein the first style vector and the second style vector are to encode style features of the simulated image and the real image;

determine a style difference between the first style vector and second style vector;

input the simulated image and the style difference to a generator neural network of the trained model; and

generate, by the generator neural network, a post-processed simulated image by applying the style difference to the simulated image.

17 . The non-transitory computer-readable medium of claim 16 , wherein the generator neural network is trained as a generative adversarial network (GAN) using the generator neural network and a discriminator neural network, wherein the generator neural network is to translate the simulated image into the post-processed simulated image based on the style difference, and wherein the discriminator neural network is to distinguish between the post-processed simulated image and the real image.

18 . The non-transitory computer-readable medium of claim 16 , wherein the style features comprise at least one of contrast, color balance, sharpness, image noise, lighting, or scene fidelity.

19 . The non-transitory computer-readable medium of claim 16 , wherein the real image comprises a first timestamp that matches a second timestamp of the simulated image.

20 . The non-transitory computer-readable medium of claim 16 , wherein the contextual data comprises metadata and label data of the real image, the metadata and label data comprising one or more of identification of objects, a pose of a sensor, map location, environment conditions, and lighting, and wherein the training simulated images are generated from the training real images by:

utilizing the metadata and the label data corresponding to the training real images to select assets corresponding to objects in the training real images, select a background corresponding to the training real images, and select lighting corresponding to the training real images;

rendering the training simulated images using the selected assets, the selected background, and the selected lighting; and

indicating the training simulated images and the training real images as the approximately-paired image pairs.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Feb 23, 2023
From: SHRIVASTAVA, ASHISH; ZHANG, CHARLES YINGJIA
To: GM CRUISE HOLDINGS LLC
Reel/Frame 062788/0344 →
Continuity (1)
Related Publication 20240282117A1 · Aug 22, 2024
References Cited (74)
US 10242292B2 · Zisimopoulos · 2019 [cited by examiner]
US 11048974B2 · Zhong · 2021 [cited by examiner]
US 11341699B1 · Gottlieb · 2022 [cited by examiner]
US 12530032B2 · Newman · 2026 [cited by examiner]
US 12530588B2 · Du · 2026 [cited by examiner]
US 20180300850A1 · Johnson · 2018 [cited by examiner]
US 20180314716A1 · Kim · 2018 [cited by examiner]
US 20180373999A1 · Xu · 2018 [cited by examiner]
US 20190042882A1 · Kim · 2019 [cited by examiner]
US 20200074674A1 · Guo · 2020 [cited by examiner]
US 20200167606A1 · Wohlhart · 2020 [cited by examiner]
US 20200311986A1 · Tong · 2020 [cited by examiner]
US 20200342276A1 · Lee · 2020 [cited by examiner]
US 20210019629A1 · Chidlovskii · 2021 [cited by examiner]
US 20210056343A1 · Toizumi · 2021 [cited by examiner]
US 20210089903A1 · Murray · 2021 [cited by examiner]
US 20210209464A1 · Bala · 2021 [cited by examiner]
US 20210215818A1 · Kim · 2021 [cited by examiner]
US 20210358115A1 · Hever · 2021 [cited by examiner]
US 20220180189A1 · Adrian · 2022 [cited by examiner]
US 20220237829A1 · Ren · 2022 [cited by examiner]
US 20220309674A1 · Zhang · 2022 [cited by examiner]
US 20230055538A1 · Miron · 2023 [cited by examiner]
US 20230114734A1 · Moon · 2023 [cited by examiner]
US 20230154165A1 · Park · 2023 [cited by examiner]
US 20230230242A1 · Lorsakul · 2023 [cited by examiner]
US 20240161444A1 · Das · 2024 [cited by examiner]
US 20240161445A1 · Fujiwaka · 2024 [cited by examiner]
US 20240185075A1 · Du · 2024 [cited by examiner]
US 20240303838A1 · Dana · 2024 [cited by examiner]
US 20240303973A1 · Ramos Dos Santos · 2024 [cited by examiner]
US 20250095112A1 · Chen · 2025 [cited by examiner]
US 20250363681A1 · Wang · 2025 [cited by examiner]
US 20260017531A1 · Lou · 2026 [cited by examiner]
I Akkaya et al., “Solving rubik's cube with a robot hand,” arXiv preprint arXiv: 1910.07113, 2019. [cited by applicant]
J. Lee et al., “Learning quadrupedal locomotion over challenging terrain,” Science robotics, 2020. [cited by applicant]
W. Wang et al., “Tartanair: A dataset to push the limits of visual slam,” in Proc. IROS, 2020. [cited by applicant]
E. Wood et al, “Fake it till you make it: Face analysis in the wild using synthetic data alone,” in Proc. ICCV, 2021. [cited by applicant]
M. Haiderbhai et al., “Robust sim2real transfer with the da vinci research kit: A study on camera, lighting, and physics domain randomization,” in Proc. IROS, 2022. [cited by applicant]
A. Shrivastava et al., “Learning from simulated and unsupervised images through adversarial training,” in Proc. CVPR, 2017. [cited by applicant]
T. Park et al., “Contrastive learning for unpaired image-to-image translation,” in Proc. ECCV, 2020. [cited by applicant]
J.-Y. Zhu et al., “Unpaired image-to-image translation using cycle-consistent adversarial networks,” in Proc. ICCV, 2017. [cited by applicant]
I. Goodfellow et al., “Generative adversarial nets,” in Proc. NeuRIPS, 2014. [cited by applicant]
A. Brock et al., “Large scale GAN training for high fidelity natural image synthesis,” in Proc. ICLR, 2019. [cited by applicant]
X. Huang et al., “Arbitrary style transfer in real-time with adaptive instance normalization,” in Proc. ICCV, 2020. [cited by applicant]
A. Karnewar et al., “MSG-GAN: Multi-scale gradients for generative adversarial networks,” in Proc. CVPR, 2020. [cited by applicant]
T. Karras et al., “Progressive growing of GANs for improved quality, stability, and variation,” in Proc. ICLR, 2018. [cited by applicant]
T. Karras et al., “A style-based generator architecture for generative adversarial networks,” in Proc. CVPR, 2019. [cited by applicant]
I. Gulrajani et al., “Improved Training of Wasserstine GAN,” in Proc. NeuRIPS, 2017. [cited by applicant]
H. Zhang et al., “Self-attention generative adversarial networks,” CoRR, abs/1805.08318, 2018. [cited by applicant]
H. Zhang et al., “Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks,” in Proc. ICCV, 2017. [cited by applicant]
H. Zhang et al., “StackGAN++: Realistic image synthesis with stacked generative adversarial networks,” CoRR, vol. abs/1710.10916, 2017. [cited by applicant]
M.-Y. Liu et al., “Coupled generative adversarial networks,” in Proc. NeuRIPS, 2016. [cited by applicant]
X. Mao et al., “Multi-class generative adversarial networks with the L2 loss function,” in Proc. ICCV, 2016. [cited by applicant]
T. Karras et al., “Alias-free generative adversarial networks,” in Proc. NeuRIPS, 2021. [cited by applicant]
P. Isola “Image-to-image translation with conditional adversarial networks,” in Proc. CVPR, 2017. [cited by applicant]
Y. Qu et al., “Enhanced pix2pix dehazing network,” in Proc. CVPR, 2019. [cited by applicant]
T. Wang et al., “High-resolution image synthesis and semantic manipulation with conditional gans,” in Proc. CVPR, 2020. [cited by applicant]
S. Kim et al., “InstaFormer: Instance-aware image-to-image translation with transformer,” in Proc. CVPR, 2022. [cited by applicant]
P. Sangkloy et al., “Scribbler: Controlling deep image synthesis with sketch and color,” in Proc. CVPR, 2017. [cited by applicant]
L. Karacan et al., “Learning to generate images of outdoor scenes from attributes and semantic layouts,” CoRR, vol. abs/1612.00215, 2016. [cited by applicant]
S. Yang et al., “Unsupervised image-to-image translation with generative prior,” in Proc. CVPR, 2022. [cited by applicant]
M. Ko et al., “Self-supervised dense consistency regularization for image-to-image translation,” in Proc. CVPR, 2022. [cited by applicant]
S. Tripathy et al., “Learning image-to-image translation using paired and unpaired training samples,” in Proc. ACCV, 2019. [cited by applicant]
A. Mustafa et al., “Transformation consistency regularization- A semi-supervised paradigm for image-to-image translation,” in Proc. ECCV, 2020. [cited by applicant]
S. Benaim et al., “One-sided unsupervised domain mapping,” in Proc. NeuRIPS, 2017. [cited by applicant]
K. He et al., “Deep residual learning for image recognition,” in Proc. CVPR, 2016. [cited by applicant]
T. Karras et al., “Analyzing and improving the image quality of StyleGAN,” in Proc. CVPR, 2020. [cited by applicant]
M. Heusel et al., “GANs trained by a two time-scale update rule converge to a nash equilibrium,” CoRR, vol. abs/1706.08500, 2017. [cited by applicant]
A. Sauer et al., “Projected GANs con-verge faster,” in Advances in Neural Information Processing Systems, 2021. [cited by applicant]
I. J. Goodfellow, “NIPS 2016 tutorial: Generative adversarial net-works,” CoRR, vol. abs/1701.00160, 2017. [cited by applicant]
D. P. Kingma et al., “Adam: A method for stochastic optimiza-tion,” in ICLR, 2015. [cited by applicant]
Y. Yazici et al., “The unusual effectiveness of averaging in GAN training,” CoRR, vol. abs/1806.04498, 2018. [cited by applicant]
Wikipedia, “Variational autoencoder”, downloaded on Jan. 24, 2023, https://en.wikipedia.org/wiki/Varionational_autoencoder, 7 pages. [cited by applicant]