IP Library Granted Patent US 11,556,875
Granted Patent B2
US 11,556,875 · App. 17/677,678 · Granted Jan 17, 2023

Generating visualizations for instructional procedures

Inventors: Abhishek Mehrotra (Milwaukee, WI); Hugo Bernardino Da Silva (Sao Paulo, BR); Richard S. Turk (Mayfield Village, OH); Timothy R. Brennan (Milwaukee, WI); Tyler L. Sheveland (Milwaukee, WI)
Assignee: Rockwell Automation Technologies, Inc.
G06Q10/06316G06F16/248G06T11/00G06T13/80G06T2200/24
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,556,875
App. No.
17/677,678
Granted
Jan 17, 2023
Kind
B2
Abstract

A system may include a processor and a memory. The memory may include computer-executable code that, when executed by the processor, causes the processor to retrieve a workflow dataset from a database based on a query input associated with an industrial automation device. The workflow dataset may include an instruction associated with one or more operations for the industrial automation device and a virtual object associated with the one or more instructions and the industrial automation device. The memory may include computer-executable code that, when executed by the processor, causes the processor to transmit a first portion of the workflow dataset to a computing device corresponding to a first instruction. The memory may include computer-executable code that, when executed by the processor, causes the processor to transmit a second portion of the workflow dataset to the computing device in response to determining that the first instruction is completed.

Claims (54)

1. A system, comprising:

a processor; and

memory comprising computer-executable code that, when executed by the processor, causes the processor to:

retrieve a workflow dataset associated with an industrial automation device, wherein the workflow dataset comprises:

one or more safety instructions associated with a user performing one or more operations on the industrial automation device, wherein the safety instructions comprise one or more preparatory steps prior to performing the one or more operations; and

one or more virtual objects associated with the one or more safety instructions and the industrial automation device;

transmit a first portion of the workflow dataset to a computing device, wherein the first portion of the workflow dataset is configured to present a first animation of a first virtual object of the one or more virtual objects via an electronic display communicably coupled to the computing device, and wherein the first animation corresponds to performing a first instruction of the one or more safety instructions;

receive image data representative of the industrial automation device from an image sensor associated with the computing device;

determine that a first operation corresponding to the first portion of the workflow dataset is performed based on the image data; and

transmit a second portion of the workflow dataset to the computing device in response to determining that the first instruction is completed by comparing the image data with an additional image data from the image sensor.

2. The system of claim 1 , wherein the one or more preparatory steps comprise a lock-out, tag-out operation.

3. The system of claim 1 , wherein the computer-executable code that, when executed by the processor, causes the processor to:

receive an input indicative of a type of the industrial automation device; and

retrieve the workflow dataset based on the type of the industrial automation device, wherein the one or more preparatory steps correspond to the type of the industrial automation device.

4. The system of claim 1 , wherein the first instruction of the one or more safety instructions corresponds to powering off the industrial automation device.

5. The system of claim 1 , wherein the second portion of the workflow dataset is configured to present a second animation of a second virtual object of the one or more virtual objects via the electronic display, and wherein the second animation corresponds to performing a second instruction of the one or more safety instructions.

6. The system of claim 5 , wherein the second instruction of the one or more safety instructions corresponds to electrically isolating the industrial automation device from an additional industrial automation device.

7. The system of claim 1 , wherein the workflow dataset comprises one or more maintenance instructions to enable the user to perform one or more maintenance operations on the industrial automation device.

8. The system of claim 7 , wherein the second portion of the workflow dataset is configured to present a second animation of a second virtual object of the one or more virtual objects via the electronic display, and wherein the second animation corresponds to performing a first maintenance instruction of the one or more maintenance instructions.

9. The system of claim 8 , wherein the second portion of the workflow dataset is transmitted in response to receiving an indication of an alarm signal, a status signal, a control signal, an event signal, a trigger signal, or any combination thereof.

10. A method, comprising:

retrieving, via a processor, a workflow dataset associated with an industrial automation device, wherein the workflow dataset comprises:

one or more safety instructions associated with a user performing one or more operations on the industrial automation device, wherein the safety instructions comprise one or more preparatory steps prior to performing the one or more operations; and

one or more virtual objects associated with the one or more safety instructions and the industrial automation device;

transmitting, via the processor, a first portion of the workflow dataset to a computing device, wherein the first portion of the workflow dataset is configured to present a first animation of a first virtual object of the one or more virtual objects via an electronic display communicably coupled to the computing device, and wherein the first animation corresponds to performing a first instruction of the one or more safety instructions;

receiving, via the processor, image data representative of the industrial automation device from an image sensor associated with the computing device;

determining, via the processor, that a first operation corresponding to the first portion of the workflow dataset is performed based on the image data; and

transmitting, via the processor, a second portion of the workflow dataset to the computing device in response to determining that the first instruction is completed by comparing the image data with an additional image data from the image sensor.

11. The method of claim 10 , comprising:

receiving, via the processor, an input from an additional computing device, wherein the input is configured to generate a visualization to be overlaid on the image data; and

transmitting, via the processor, the visualization to the computing device, wherein the computing device is configured to present the visualization overlaid on the image data via the electronic display.

12. The method of claim 11 , wherein the visualization comprises a geometric image.

13. The method of claim 10 , comprising:

receiving, via the processor, a notification of an operation of the industrial automation device that has deviated from a desired operation; and

transmitting, via the processor, the second portion of the workflow dataset in response to receiving the notification.

14. The method of claim 10 , wherein determining, via the processor, that the first operation corresponding to the first portion of the workflow dataset is performed based on the image data comprises comparing the image data to expected image data indicative of the first portion of the workflow dataset being performed.

15. The method of claim 10 , comprising:

receiving, via the processor, an input indicative of a type of the industrial automation device; and

retrieving, via the processor, the workflow dataset based on the type of the industrial automation device, wherein the one or more preparatory steps are particular to the industrial automation device.

16. A tangible, non-transitory, computer-readable medium comprising instructions that, when executed by a processor, cause a control system to perform operations comprising:

receiving a query input associated with an industrial automation device;

retrieving a workflow dataset, wherein the workflow dataset comprises:

one or more safety instructions associated with a user performing one or more operations on the industrial automation device, wherein the safety instructions comprise one or more preparatory steps prior to performing the one or more operations; and

one or more virtual objects associated with the one or more safety instructions and the industrial automation device;

transmitting a first portion of the workflow dataset to a computing device, wherein the first portion of the workflow dataset is configured to present a first animation of a first virtual object of the one or more virtual objects via an electronic display communicably coupled to the computing device, and wherein the first animation corresponds to performing a first instruction of the one or more safety instructions;

receiving image data representative of the industrial automation device from an image sensor associated with the computing device;

determining that a first operation corresponding to the first portion of the workflow dataset is performed based on the image data; and

transmitting a second portion of the workflow dataset to the computing device in response to determining that the first instruction is completed by comparing the image data with an additional image data from the image sensor.

17. The tangible, non-transitory, computer-readable medium of claim 16 , the operations comprising:

receiving an input indicative of a type of the industrial automation device; and

retrieving the workflow dataset based on the type of the industrial automation device, wherein the one or more preparatory steps are particular to the industrial automation device.

18. The tangible, non-transitory, computer-readable medium of claim 16 , wherein the one or more preparatory steps comprise a lock-out, tag-out operation.

19. The tangible, non-transitory, computer-readable medium of claim 16 , wherein the first instruction of the one or more safety instructions corresponds to powering off the industrial automation device.

20. The tangible, non-transitory, computer-readable medium of claim 16 , wherein the second portion of the workflow dataset is configured to present a second animation of a second virtual object of the one or more virtual objects, and wherein the second animation corresponds to performing a second instruction of the one or more safety instructions.

Continuity (2)
Continuation 16687462 · Nov 18, 2019
Related Publication 20220180283A1 · Jun 9, 2022