IP Library Granted Patent US 12682776
Granted Patent B2
US 12682776 · App. 17/823,188 · Granted Jul 14, 2026

Mixed reality scenario generation for cross-industry training

Inventors: Mary A Mt. Pleasant (Tyngsboro, MA); Tushar Agrawal (West Fargo, ND); Sarbajit K. Rakshit (Kolkata, IN); Jeremy R. Fox (Georgetown, TX)
Assignee: International Business Machines Corporation
G09B5/02
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12682776
App. No.
17/823,188
Filed
Aug 30, 2022
Granted
Jul 14, 2026
Kind
B2
Art Unit
3715
USPC
434/PCA.024
Abstract

In an approach to mixed reality scenario generation for cross industry training, a computer receives data associated with tasks performed by a user. A computer identifies an activity in a first industry associated with the tasks. A computer determines that one or more activities associated with a job in a second industry are similar to the activity in the first industry. A computer generates a mixed reality scenario representing a virtual work environment associated with the second industry using mixed reality. A computer generates the one or more activities associated with the job in the second industry in the mixed reality scenario. A computer displays the virtual work environment associated with the second industry and the one or more activities associated with the job in the second industry to the user in a device for viewing a mixed reality scenario while the user performs the activity in the first industry.

Claims (69)

1 . A computer-implemented method comprising:

receiving, by one or more computer processors, first sensor data from one or more sensor devices located in a work environment associated with a first industry, wherein the one or more sensor devices are positioned in the work environment to enable features of the work environment to be captured in the first sensor data;

analyzing, by one or more computer processors, the first sensor data to identify the work environment associated with the first industry based on the features captured in the first sensor data and match the features with one or more tasks performed within the work environment of the first industry;

identifying, by one or more computer processors, an activity in the first industry associated with the work environment and the one or more tasks;

determining, by one or more computer processors, that one or more activities associated with a job in a second industry correspond to the activity in the first industry;

generating, by one or more computer processors, a mixed reality scenario representing a virtual work environment associated with the second industry using mixed reality technology;

generating, by one or more computer processors, the one or more activities associated with the job in the second industry in the mixed reality scenario, wherein the one or more activities are generated to allow user performance of the one or more activities to be detected by at least one sensor device;

displaying, by one or more computer processors, the virtual work environment associated with the second industry and the one or more activities associated with the job in the second industry to a user in a device for viewing the mixed reality scenario while the user performs the activity in the first industry; and

responsive to at least one sensor device detecting user performance of the one or more activities associated with the job in the second industry, evaluating in real time, by one or more computer processors, the user performance of the one or more activities using second sensor data generated by the at least one sensor device.

2 . The computer-implemented method of claim 1 , wherein the first and second sensor data is received from sensor devices selected from a group consisting of: a gaze point tracker, an imaging device, a microphone, environmental sensor, a global positioning system (GPS) device, an accelerometer, a pressure sensor, and a biometric sensor.

3 . The computer-implemented method of claim 1 , wherein determining that the one or more activities associated with the job in the second industry correspond to the activity in the first industry includes comparing one or more movements of a user, comparing one or more required tools, and comparing one or more required physical attributes of the user.

4 . The computer-implemented method of claim 1 , wherein generating the virtual work environment associated with the job in the second industry is performed using a generative adversarial network (GAN) module.

5 . The computer-implemented method of claim 1 , wherein evaluating user performance of the one or more activities further comprises:

storing, by one or more computer processors, the second sensor data associated with the performance of the one or more activities associated with the job in the second industry;

determining, by one or more computer processors, the performance of the one or more activities associated with the job in the second industry is acceptable; and

notifying, by one or more computer processors, the user that training for the job in the second industry is complete.

6 . The computer-implemented method of claim 1 , further comprising:

comparing, by one or more computer processors, the one or more tasks associated with the first activity of the first industry and the one or more tasks associated with the second activity of the second industry;

based on the comparing, identifying, by one or more computer processors, one or more similarities between the one or more tasks associated with the first activity of the first industry and the one or more tasks associated with the second activity of the second industry; and

storing, by one or more computer processors, the one or more similarities between the one or more tasks associated with the first activity of the first industry and the one or more tasks associated with the second activity of the second industry.

7 . The computer-implemented method of claim 6 , wherein identifying the one or more similarities between the one or more tasks associated with the first activity of the first industry and the one or more tasks associated with the second activity of the second industry further comprises:

determining, by one or more computer processors, a degree of similarity between two tasks by calculating a similarity score between at least one of the one or more tasks associated with the first activity of the first industry and at least one of the one or more tasks associated with the second activity of the second industry based on a pre-defined number of attributes associated with each task.

8 . A computer program product comprising:

one or more computer readable storage media and program instructions collectively stored on the one or more computer readable storage media, the stored program instructions comprising:

program instructions to receive first sensor data from one or more sensor devices located in a work environment associated with a first industry, wherein the one or more sensor devices are positioned in the work environment to enable features of the work environment to be captured in the first sensor data;

program instructions to analyze the first sensor data to identify the work environment associated with the first industry based on the features captured in the first sensor data and match the features with one or more tasks performed within the work environment of the first industry;

program instructions to identify an activity in the first industry associated with the work environment and the one or more tasks;

program instructions to determine that one or more activities associated with a job in a second industry correspond to the activity in the first industry;

program instructions to generate a mixed reality scenario representing a virtual work environment associated with the second industry using mixed reality technology;

program instructions to generate the one or more activities associated with the job in the second industry in the mixed reality scenario, wherein the one or more activities are generated to allow user performance of the one or more activities to be detected by at least one sensor device;

program instructions to display the virtual work environment associated with the second industry and the one or more activities associated with the job in the second industry to the user in a device for viewing the mixed reality scenario while the user performs the activity in the first industry; and

responsive to the at least one sensor device detecting user performance of the one or more activities associated with the job in the second industry, program instructions to evaluate in real time the user performance of the one or more activities using second sensor data generated by the at least one sensor device.

9 . The computer program product of claim 8 , wherein the first and second sensor data is received from sensor devices selected from a group consisting of: a gaze point tracker, an imaging device, a microphone, environmental sensor, a global positioning system (GPS) device, an accelerometer, a pressure sensor, and a biometric sensor.

10 . The computer program product of claim 8 , wherein the stored program instructions to determine that the one or more activities associated with the job in the second industry correspond to the activity in the first industry includes comparing one or more movements of the user, comparing one or more required tools, and comparing one or more required physical attributes of the user.

11 . The computer program product of claim 8 , wherein the stored program instructions to generate the virtual work environment associated with the job in the second industry use a generative adversarial network (GAN) module.

12 . The computer program product of claim 8 , wherein the stored program instructions to evaluate user performance further comprise:

program instructions to store the sensor activity data associated with the performance of the one or more activities associated with the job in the second industry;

program instructions to determine the performance of the one or more activities associated with the job in the second industry is acceptable; and

program instructions to notify the user that training for the job in the second industry is complete.

13 . The computer program product of claim 8 , the stored program instructions further comprising:

program instructions to compare the one or more tasks associated with the first activity of the first industry and the one or more tasks associated with the second activity of the second industry;

program instructions to identify one or more similarities between the one or more tasks associated with the first activity of the first industry and the one or more tasks associated with the second activity of the second industry based on the comparing of the one or more tasks associated with the first activity of the first industry and the one or more tasks associated with the second activity of the second industry; and

program instructions to store the one or more similarities between the one or more tasks associated with the first activity of the first industry and the one or more tasks associated with the second activity of the second industry.

14 . The computer program product of claim 13 , wherein the program instructions to identify the one or more similarities between the one or more tasks associated with the first activity of the first industry and the one or more tasks associated with the second activity of the second industry comprise:

program instructions to determine a degree of similarity between two tasks by calculating a similarity score between at least one of the one or more tasks associated with the first activity of the first industry and at least one of the one or more tasks associated with the second activity of the second industry based on a pre-defined number of attributes associated with each task.

15 . A computer system comprising:

one or more computer processors;

one or more computer readable storage media;

program instructions collectively stored on the one or more computer readable storage media for execution by at least one of the one or more computer processors, the stored program instructions comprising:

program instructions to receive first sensor data from one or more sensor devices located in a work environment associated with a first industry, wherein the one or more sensor devices are positioned in the work environment to enable features of the work environment to be captured in the first sensor data;

program instructions to analyze the first sensor data to identify the work environment associated with the first industry based on the features captured in the first sensor data and match the features with one or more tasks performed within the work environment of the first industry;

program instructions to identify an activity in the first industry associated with the work environment and the one or more tasks;

program instructions to determine that one or more activities associated with a job in a second industry correspond to the activity in the first industry;

program instructions to generate a mixed reality scenario representing a virtual work environment associated with the second industry using mixed reality technology;

program instructions to generate the one or more activities associated with the job in the second industry in the mixed reality scenario, wherein the one or more activities are generated to allow user performance of the one or more activities to be detected by at least one sensor device;

program instructions to display the virtual work environment associated with the second industry and the one or more activities associated with the job in the second industry to the user in a device for viewing the mixed reality scenario while the user performs the activity in the first industry; and

responsive to the at least one sensor device detecting user performance of the one or more activities associated with the job in the second industry, program instructions to evaluate in real time the user performance of the one or more activities using second sensor data generated by the at least one sensor device.

16 . The computer system of claim 15 , wherein the stored program instructions to determine that the one or more activities associated with the job in the second industry correspond to the activity in the first industry includes comparing one or more movements of the user, comparing one or more required tools, and comparing one or more required physical attributes of the user.

17 . The computer system of claim 15 , wherein the stored program instructions to generate the virtual work environment associated with the job in the second industry use a generative adversarial network (GAN) module.

18 . The computer system of claim 15 , wherein the stored program instructions to evaluate user performance further comprise:

program instructions to store the sensor activity data associated with the performance of the one or more activities associated with the job in the second industry;

program instructions to determine the performance of the one or more activities associated with the job in the second industry is acceptable; and

program instructions to notify the user that training for the job in the second industry is complete.

19 . The computer system of claim 15 , the stored program instructions further comprising:

program instructions to compare the one or more tasks associated with the first activity of the first industry and the one or more tasks associated with the second activity of the second industry;

program instructions to identify one or more similarities between the one or more tasks associated with the first activity of the first industry and the one or more tasks associated with the second activity of the second industry based on the comparing of the one or more tasks associated with the first activity of the first industry and the one or more tasks associated with the second activity of the second industry; and

program instructions to store the one or more similarities between the one or more tasks associated with the first activity of the first industry and the one or more tasks associated with the second activity of the second industry.

20 . The computer system of claim 19 , wherein the program instructions to identify the one or more similarities between the one or more tasks associated with the first activity of the first industry and the one or more tasks associated with the second activity of the second industry comprise:

program instructions to determine a degree of similarity between two tasks by calculating a similarity score between at least one of the one or more tasks associated with the first activity of the first industry and at least one of the one or more tasks associated with the second activity of the second industry based on a pre-defined number of attributes associated with each task.