Granted Patent B2

US 12,548,462 · App. 18/053,347 · Granted Feb 10, 2026

Inventors: Mak Hoi Victor Hau (Markham, CA); Shikhar Kwatra (San Jose, CA); Devon Deane (Toronto, CA); Xu Wang (Markham, CA); Minhua Zhu (Toronto, CA)

Assignee: International Business Machines Corporation

G09B9/052G06T19/006G06V10/764G06V10/774G06V10/82G06V20/20G06V20/56

Abstract

Claims (64)

1 . A computer program product for generating an augmented environment for a user while operating a vehicle, the computer program product comprising a computer readable storage medium having computer readable program code embodied therein that is executable to perform operations, the operations comprising:

receiving a view environment image of a user field of view of a user operating the vehicle;

inputting the view environment image and a user difficulty level for the user into a generator, comprising a neural network model, to generate an augmented environment image including virtual objects added to the view environment image; and

transmitting the virtual objects to a mixed reality display to render in the user field of view;

receiving information, including user biometrics, gathered during the user operating the vehicle to avoid the virtual objects rendered in the user field of view;

modifying the user difficulty level to produce a modified user difficulty level based on the received information; and

inputting the modified user difficulty level to the generator to generate a further augmented environment image with added virtual objects to transmit to the mixed reality display to further render in the user field of view.

2 . The computer program product of claim 1 , wherein the operations further comprise:

inputting a specified user difficulty level and training environment images into the generator to output training augmented environment images including added virtual objects;

determining whether the training augmented environment images and the added virtual objects are fake or real;

training the generator to output a training augmented environment image with added virtual objects with a low confidence level in response to determining the training augmented environment image is fake; and

training the generator to output the training augmented environment image with the added virtual objects with a high confidence level in response to determining the training augmented environment images are real.

3 . The computer program product of claim 2 , wherein a discriminator, comprising a neural network, performs the determining whether the training augmented environment images and the added virtual objects are fake or real, wherein the operations further comprise:

training the discriminator to classify real environment images for a specified user difficulty level as real for the user difficulty level; and

training the discriminator to classify training augmented environment images from the generator for a specified user difficulty level as fake for the user difficulty level.

4 . The computer program product of claim 3 , wherein the generator and the discriminator are implemented in a Conditional Generative Adversarial Network (GAN).

5 . The computer program product of claim 1 , wherein the virtual objects are generated based on the user difficulty level for the user operating the vehicle to output the virtual objects to provide when rendered in the user field of view a level of obstacles consistent with the user difficulty level.

6 . The computer program product of claim 1 , wherein the operations further comprise:

determining a user performance in operating the vehicle to avoid the virtual objects rendered in the user field of view; and

determining a comfort level of the user in operating the vehicle to avoid the virtual objects rendered in the user field of view based on the user biometrics and the user performance, where the user difficulty level is modified to produce the modified user difficulty level based on the comfort level.

7 . The computer program product of claim 1 , wherein the view environment image is generated by at least one camera affixed to the vehicle the user is operating to capture images in the user field of view while operating the vehicle.

8 . The computer program product of claim 1 , wherein the virtual objects rendered in the user field of view by the mixed reality display comprise virtual traffic obstacles for the user to avoid while operating the vehicle.

9 . A system for generating an augmented environment for a user while operating a vehicle, comprising:

a processor; and

a computer readable storage medium having computer readable program code embodied therein that when executed by the processor performs operations, the operations comprising:

receiving a view environment image of a user field of view of a user operating the vehicle;

inputting the view environment image and a user difficulty level for the user into a generator, comprising a neural network model, to generate an augmented environment image including virtual objects added to the view environment image;

transmitting the virtual objects to a mixed reality display to render in the user field of view;

receiving information, including user biometrics, gathered during the user operating the vehicle to avoid the virtual objects rendered in the user field of view;

modifying the user difficulty level to produce a modified user difficulty level based on the received information; and

inputting the modified user difficulty level to the generator to generate a further augmented environment image with added virtual objects to transmit to the mixed reality display to further render in the user field of view.

10 . The system of claim 9 , wherein the operations further comprise:

inputting a specified user difficulty level and training environment images into the generator to output training augmented environment images including added virtual objects;

determining whether the training augmented environment images and the added virtual objects are fake or real;

training the generator to output a training augmented environment image with added virtual objects with a low confidence level in response to determining the training augmented environment image is fake; and

training the generator to output the training augmented environment image with the added virtual objects with a high confidence level in response to determining the training augmented environment images are real.

11 . The system of claim 10 , wherein a discriminator, comprising a neural network, performs the determining whether the training augmented environment images and the added virtual objects are fake or real, wherein the operations further comprise:

training the discriminator to classify real environment images for a specified user difficulty level as real for the user difficulty level; and

training the discriminator to classify training augmented environment images from the generator for a specified user difficulty level as fake for the user difficulty level.

12 . The system of claim 9 , wherein the virtual objects are generated based on the user difficulty level for the user operating the vehicle to output the virtual objects to provide when rendered in the user field of view a level of obstacles consistent with the user difficulty level.

13 . The system of claim 9 , wherein the operations further comprise:

determining a user performance in operating the vehicle to avoid the virtual objects rendered in the user field of view; and

determining a comfort level of the user in operating the vehicle to avoid the virtual objects rendered in the user field of view based on the user biometrics and the user performance, wherein the user difficulty level is modified to produce a modified user difficulty level based on the comfort level.

14 . The system of claim 9 , wherein the virtual objects rendered in the user field of view by the mixed reality display comprise virtual traffic obstacles for the user to avoid while operating the vehicle.

15 . A method for generating an augmented environment for a user while operating a vehicle, comprising:

receiving a view environment image of a user field of view of a user operating the vehicle;

inputting the view environment image and a user difficulty level for the user into a generator, comprising a neural network model, to generate an augmented environment image including virtual objects added to the view environment image; and

transmitting the virtual objects to a mixed reality display to render in the user field of view;

receiving information, including user biometrics, gathered during the user operating the vehicle to avoid the virtual objects rendered in the user field of view;

modifying the user difficulty level to produce a modified user difficulty level based on the received information; and

inputting the modified user difficulty level to the generator to generate a further augmented environment image with added virtual objects to transmit to the mixed reality display to further render in the user field of view.

16 . The method of claim 15 , further comprising:

inputting a specified user difficulty level and training environment images into the generator to output training augmented environment images including added virtual objects;

determining whether the training augmented environment images and the added virtual objects are fake or real;

training the generator to output a training augmented environment image with added virtual objects with a low confidence level in response to determining the training augmented environment image is fake; and

training the generator to output the training augmented environment image with the added virtual objects with a high confidence level in response to determining the training augmented environment images are real.

17 . The method of claim 16 , wherein a discriminator, comprising a neural network, performs the determining whether the training augmented environment images and the added virtual objects are fake or real, further comprising:

training the discriminator to classify real environment images for a specified user difficulty level as real for the user difficulty level; and

training the discriminator to classify training augmented environment images from the generator for a specified user difficulty level as fake for the user difficulty level.

18 . The method of claim 15 , wherein the virtual objects are generated based on the user difficulty level for the user operating the vehicle to output the virtual objects to provide when rendered in the user field of view a level of obstacles consistent with the user difficulty level.

19 . The method of claim 15 , further comprising:

determining a user performance in operating the vehicle to avoid the virtual objects rendered in the user field of view; and

determining a comfort level of the user in operating the vehicle to avoid the virtual objects rendered in the user field of view based on the user biometrics and the user performance, wherein the user difficulty level is modified to produce a modified user difficulty level based on the comfort level.

20 . The method of claim 15 , wherein the virtual objects rendered in the user field of view by the mixed reality display comprise virtual traffic obstacles for the user to avoid while operating the vehicle.

Assignments (1)

Continuity (1)

References Cited (34)