IP Library Granted Patent US 9,947,138
Granted Patent B2
US 9,947,138 · App. 14/686,427 · Granted Apr 17, 2018

System and method for augmented reality display of dynamic environment information

Inventors: Brian Bare (Newport News, VA); Jonathan Martin (Newport News, VA); Patrick Ryan (Newport News, VA); Paul Sells (Los Angeles, CA); Mark Lawrence (Newport News, VA); Justin Brooks (Newport News, VA); Joshua Mason (Newport News, VA)
Assignee: Huntington Ingalls Incorporated
G06T19/006G08B21/12H04W4/02
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 9,947,138
App. No.
14/686,427
Granted
Apr 17, 2018
Kind
B2
Abstract

A method for providing environment information to a mobile device user is presented. The method comprises receiving a request for target environment information from a mobile device, determining the pose of the mobile interface device relative to the target environment, and obtaining target environment data for one or more measurable environment parameters (e.g., radiation level). The target environment data is used to assemble augmented reality information configured for viewing in conjunction with a real-time view of the target environment captured by the mobile interface device. The target augmented reality information is then transmitted to the mobile device for display to the user.

Claims (49)

1. A method for providing environment information to a mobile interface device user in a dynamic structural environment in the form of a dynamic structure having an associated environment coordinate system, the method comprising:

receiving on a central data processor from the mobile interface device over a communication network a request for target environment information, the request including request information usable by the central data processor to determine the pose of the mobile interface device relative to a target environment that is a portion of the dynamic structural environment;

receiving mobile interface device location information from a local positioning system of the dynamic structure, said location information being provided using the environment coordinate system;

determining the pose of the mobile interface device relative to the target environment using the request information and the mobile interface device location information;

obtaining target environment data for one or more measurable environment parameters;

assembling augmented reality (AR) environment information for transmission to and display on the mobile interface device, the AR environment information being assembled using the target environment data and being configured for viewing in conjunction with a real-time view of the target environment captured by the mobile interface device; and

transmitting the AR environment information to the mobile interface device,

wherein the action of determining the pose of the mobile interface device accounts for spatial changes in the environment coordinate system in the target environment due to changes in the dynamic structural environment.

2. A method according to claim 1 , wherein the mobile interface device is one of the set consisting of a tablet computer, a smartphone, and a wearable heads-up display.

3. A method according to claim 1 wherein the request for target environment information includes a digital image of the target area captured by the mobile interface device.

4. A method according to claim 1 wherein the action of obtaining target environment data includes:

receiving environment parameter data from one or more sensors located in or adjacent the target environment.

5. A method according to claim 4 wherein the AR environment information includes a graphical representation of at least one of the one or more measurable environment parameters, the graphical representation having been constructed using the target environment data.

6. A method according to claim 1 wherein the target environment data comprises radiation sensor data from at least one location in or adjacent the target environment.

7. A method according to claim 6 further comprising:

constructing a three dimensional estimate of radiation levels in the target environment.

8. A method according to claim 7 wherein the AR environment information includes a graphical representation of the three dimensional estimate of radiation levels in the target environment that can be superimposed over the real-time view of the target environment captured by the mobile interface device.

9. A method according to claim 1 wherein the actions of determining, obtaining, assembling and transmitting are periodically repeated.

10. An automated system for providing environment information to a mobile interface device in a dynamic structural environment in the form of a dynamic structure having an environment coordinate system, the system comprising:

at least one mobile interface device configured for variable disposition within the dynamic structural environment, capturing information about a target environment within the structural environment, and transmitting the target area information over a network;

a domain coordinate management system comprising one or more data processors configured for receiving information on a current conformation of the dynamic structural environment and resolving spatial differences in the environment coordinate system between the current conformation of the dynamic structural environment and a stored model of the dynamic structural environment;

a local positioning system disposed at least partially within the dynamic structure, the local positioning system being in communication with the at least one mobile interface device and the domain coordinate system via the network and being configured to provide location information usable to determine a mobile interface device pose, said mobile interface device pose comprising a position and orientation of the mobile interface device relative to the target environment in the environment coordinate system; and

a central processor comprising at least one data processing machine in communication with the at least one mobile interface device, the domain coordinate management system, and the local positioning system via the network, the central processor being configured for

receiving from a requesting one of the at least one mobile interface device a request for target environment information, the request including information usable to determine the pose of the requesting mobile interface device relative to the target environment,

determining the pose of the requesting mobile interface device relative to the target environment, accounting for spatial changes in the environment coordinate system in the target area due to changes in the dynamic structural environment,

obtaining target environment data for one or more measurable environment parameters,

assembling augmented reality (AR) environment information for transmission to and display on the requesting mobile interface device, the AR environment information being assembled using the target environment data and being configured for viewing in conjunction with a real-time view of the target environment captured by the requesting mobile interface device, and

transmitting the AR environment information to the requesting mobile interface device.

11. A system according to claim 10 further comprising:

at least one environment data system comprising one or more sensors located in or adjacent the target environment, the one or more sensors being configured for capturing the target environment data.

12. A system according to claim 11 wherein the one or more sensors are radiation sensors and the target environment data comprises a radiation level from at least one location in or adjacent the target environment.

13. A system according to claim 10 wherein the domain coordinate management system comprises one or more computers in communication with the central processor and the local positioning system and is being configured for

establishing and maintaining spatial addresses for spaces and points within the dynamic structural environment, and

receiving spatial address information from both the local positioning system and from the stored model of the dynamic structural environment.

14. A method for providing radiation environment information to a mobile interface device user in a dynamic structural environment in the form of a dynamic structure having an associated environment coordinate system, the method comprising:

receiving on a central data processor from the mobile interface device over a communication network a request for target area radiation information, the request including request information usable by the central data processor to determine a pose of the mobile interface device relative to a target area in the dynamic structural environment;

receiving mobile interface device location information from a local positioning system of the dynamic structure, said location information being provided using the environment coordinate system;

determining the pose of the mobile interface device relative to the target area using the request information and the mobile interface device location information;

obtaining target area radiation information;

assembling augmented reality (AR) radiation environment information for transmission to and display on the mobile interface device, the AR radiation environment information being assembled using the target area radiation information and being configured for viewing in conjunction with a real-time view of the target area captured by the mobile interface device; and

transmitting the AR radiation environment information to the mobile interface device,

wherein the action of determining the pose of the mobile interface device accounts for spatial changes in the environment coordinate system in the target area due to changes in the dynamic structural environment.

15. A method according to claim 14 , wherein the mobile interface device is one of the set consisting of a tablet computer, a smartphone, and a wearable heads-up display.

16. A method according to claim 14 wherein the request for target environment information includes a digital image of the target area captured by the mobile interface device.

17. A method according to claim 14 wherein the action of obtaining radiation data includes:

receiving radiation data from one or more radiation sensors located in or adjacent the target environment.

18. A method according to claim 17 further comprising:

constructing a three dimensional estimate of radiation levels in the target area.

19. A method according to claim 18 wherein the AR radiation environment information includes a graphical representation of the three dimensional estimate of radiation levels in the target area that can be superimposed over the real-time view of the target area captured by the mobile interface device.

Assignments (4)
RELEASE OF SECURITY INTEREST Recorded Dec 7, 2017
From: JPMORGAN CHASE BANK, N.A., AS COLLATERAL AGENT
To: HUNTINGTON INGALLS INCORPORATED (F/K/A NORTHROP GRUMMAN SHIPBUILDING, INC.)
Reel/Frame 044325/0332 →
PATENT SECURITY AGREEMENT Recorded Jul 13, 2015
From: HUNTINGTON INGALLS INCORPORATED
To: JPMORGAN CHASE BANK, N.A., AS COLLATERAL AGENT
Reel/Frame 036099/0065 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 1, 2015
From: BARE, BRIAN; MARTIN, JONATHAN; RYAN, PATRICK; LAWRENCE, MARK; BROOKS, JUSTIN; MASON, JOSHUA
To: HUNTINGTON INGALLS, INC.
Reel/Frame 035755/0980 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 1, 2015
From: SELLS, PAUL
To: HUNTINGTON INGALLS, INC.
Reel/Frame 035756/0138 →
Continuity (6)
Provisional Application 61995610 · Apr 15, 2014
Provisional Application 61984159 · Apr 25, 2014
Provisional Application 61997793 · Jun 9, 2014
Provisional Application 62025041 · Jul 16, 2014
Provisional Application 62036762 · Aug 13, 2014
Related Publication 20150294506A1 · Oct 15, 2015