IP Library › Granted Patent US 10,878,240
Granted Patent B2
US 10,878,240 · App. 15/898,502 · Granted Dec 29, 2020

Augmented reality user interface on mobile device for presentation of information related to industrial process, control and automation system, or other system

Inventors: Graeme Laycock (Hunters Hill, AU); Peter Horsley (Wahroonga, AU); Rohan McAdam (Yetholme, AU)
Assignee: Honeywell International Inc.
G06K9/00671G06K9/2081G06K9/46G06K2209/19
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,878,240
App. No.
15/898,502
Filed
Feb 17, 2018
Granted
Dec 29, 2020
Kind
B2
Art Unit
2663
USPC
382/291
Abstract

A method includes identifying one or more first devices contained in at least one image captured by a mobile device. The method also includes obtaining data associated with at least one of: the one or more first devices and one or more second devices connected to or interacting with the one or more first devices. The method further includes presenting information overlaid over the at least one image by the mobile device to a user, where the presented information includes the data or information based on the data. The presented information varies based on perceived distance or proximity of the mobile device to the one or more first devices.

Claims (70)

1. A method for presenting information relating to an industrial process of a control and automation system on a mobile device using an augmented reality user interface comprising:

identifying one or more first devices contained in at least one image captured by the mobile device;

obtaining data associated with at least one of: the one or more first devices and one or more second devices connected to or interacting with the one or more first devices; and

presenting information comprising different augmented reality objects overlaid over the at least one image by the mobile device to a user, the presented information comprising the data or information based on the data;

wherein the presented information includes various levels of information that varies based on perceived distance or proximity of the mobile device to the one or more first devices which is based on at least one of actual distance, proximity, or a size of the one or more first devices within the at least one image, and

wherein the augmented reality objects comprise:

a name and a status of multiple pieces of equipment of the control and automation system overlaid over the at least one image when the mobile device is farther away from the one or more first devices; and

statuses of individual modules in at least one of the pieces of equipment of the control and automation system overlaid over the at least one image when the mobile device is closer to the one or more first devices.

2. The method of claim 1 , wherein identifying the one or more first devices comprises at least one of:

detecting the user dwelling on a visual identifier associated with at least one of the one or more first devices and identifying the at least one of the one or more first devices using the visual identifier; and

detecting the user dwelling on the at least one of the one or more first devices.

3. The method of claim 1 , wherein obtaining the data comprises at least one of:

receiving the data directly from at least one of the one or more first devices and the one or more second devices; and

receiving the data indirectly from at least one of the one or more first devices and the one or more second devices via another component.

4. The method of claim 1 , wherein presenting the information comprises:

presenting different augmented reality objects over the at least one image based on the perceived distance or proximity of the mobile device to the one or more first devices.

5. The method of claim 1 , wherein the augmented reality objects further comprise at least one of:

one or more issues affecting at least one of the pieces of equipment overlaid over the at least one image; and

a window containing data associated with at least one of the modules in at least one of the pieces of equipment overlaid over the at least one image.

6. The method of claim 1 , wherein presenting the information comprises:

presenting at least some of the information in a window; and

allowing the user to navigate within the window and to select additional information for display in the window.

7. The method of claim 1 , wherein obtaining the data comprises:

detecting the user dwelling on an augmented reality object overlaid over the at least one image; and

obtaining information associated with the augmented reality object.

8. A mobile device for presenting information relating to an industrial process of a control and automation system on the mobile device using an augmented reality user interface comprising:

a display;

a camera configured to capture at least one image of one or more first devices; and

at least one processor configured to:

identify the one or more first devices contained in the at least one image;

obtain data associated with at least one of: the one or more first devices and one or more second devices connected to or interacting with the one or more first devices; and

present information comprising different augmented reality objects overlaid over the at least one image on the display to a user, the presented information comprising the data or information based on the data;

wherein the presented information includes various levels of information that varies based on perceived distance or proximity of the mobile device to the one or more first devices which is based on at least one of actual distance, proximity, or a size of the one or more first devices within the at least one image, and

wherein the augmented reality objects comprise:

a name and a status of multiple pieces of equipment of the control and automation system overlaid over the at least one image when the mobile device is farther away from the one or more first devices; and

statuses of individual modules in at least one of the pieces of equipment of the control and automation system overlaid over the at least one image when the mobile device is closer to the one or more first devices.

9. The mobile device of claim 8 , wherein, to identify the one or more first devices, the at least one processor is configured to:

detect the user dwelling on a visual identifier associated with at least one of the one or more first devices and identify the at least one of the one or more first devices using the visual identifier; and

detect the user dwelling on the at least one of the one or more first devices.

10. The mobile device of claim 8 , wherein, to present the information, the at least one processor is configured to present different augmented reality objects over the at least one image based on the perceived distance or proximity of the mobile device to the one or more first devices.

11. The mobile device of claim 8 , wherein the augmented reality objects further comprise at least one of:

one or more issues affecting at least one of the pieces of equipment overlaid over the at least one image; and

a window containing data associated with at least one of the modules in at least one of the pieces of equipment overlaid over the at least one image.

12. The mobile device of claim 8 , wherein, to present the information, the at least one processor is configured to:

present at least some of the information in a window; and

allow the user to navigate within the window and to select additional information for display in the window.

13. The mobile device of claim 8 , wherein, to obtain the data, the at least one processor is configured to:

detect the user dwelling on an augmented reality object overlaid over the at least one image; and

obtain information associated with the augmented reality object.

14. A non-transitory computer readable medium containing instructions that when executed cause at least one processing device, for presenting information relating to an industrial process of a control and automation system on a mobile device using an augmented reality user interface, to:

identify one or more first devices contained in at least one image captured by a mobile device;

obtain data associated with at least one of: the one or more first devices and one or more second devices connected to or interacting with the one or more first devices; and

present information comprising different augmented reality objects overlaid over the at least one image by the mobile device to a user, the presented information comprising the data or information based on the data;

wherein the presented information includes various levels of information that varies based on perceived distance or proximity of the mobile device to the one or more first devices which is based on at least one of actual distance, proximity, or a size of the one or more first devices within the at least one image, and

wherein the augmented reality objects comprise:

a name and a status of multiple pieces of equipment of the control and automation system overlaid over the at least one image when the mobile device is farther away from the one or more first devices; and

statuses of individual modules in at least one of the pieces of equipment of the control and automation system overlaid over the at least one image when the mobile device is closer to the one or more first devices.

15. The non-transitory computer readable medium of claim 14 , wherein the instructions that when executed cause the at least one processing device to identify the one or more first devices comprise:

instructions that when executed cause the at least one processing device to:

detect the user dwelling on a visual identifier associated with at least one of the one or more first devices and identify the at least one of the one or more first devices using the visual identifier; and

detect the user dwelling on the at least one of the one or more first devices.

16. The non-transitory computer readable medium of claim 14 , wherein the instructions that when executed cause the at least one processing device to present the information comprise:

instructions that when executed cause the at least one processing device to present different augmented reality objects over the at least one image based on the perceived distance or proximity of the mobile device to the one or more first devices.

17. The non-transitory computer readable medium of claim 14 , wherein the augmented reality objects comprise at least one of:

one or more issues affecting at least one of the pieces of equipment overlaid over the at least one image; and

a window containing data associated with at least one of the modules in at least one of the pieces of equipment overlaid over the at least one image.

18. The non-transitory computer readable medium of claim 14 , wherein the instructions that when executed cause the at least one processing device to present the information comprise:

instructions that when executed cause the at least one processing device to:

present at least some of the information in a window; and

allow the user to navigate within the window and to select additional information for display in the window.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Feb 17, 2018
From: LAYCOCK, GRAEME; HORSLEY, PETER; MCADAM, ROHAN
To: HONEYWELL INTERNATIONAL INC.
Reel/Frame 045358/0118 →
Continuity (2)
Provisional Application 62521626 · Jun 19, 2017
Related Publication 20180365495A1 · Dec 20, 2018
Cited By (2)
US 12,307,550 US 12,387,384