IP Library Granted Patent US 10,497,177
Granted Patent B1
US 10,497,177 · App. 15/709,115 · Granted Dec 3, 2019

Tool for onsite augmentation of reality meshes

Inventors: Stéphane Côté (Lac Beauport, CA); Maxime Ménard (Quebec, CA)
Assignee: Bentley Systems, Incorporated
G06T19/006G06T17/20G06T19/003G06F17/50
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,497,177
App. No.
15/709,115
Granted
Dec 3, 2019
Kind
B1
Abstract

In one embodiment, augmented reality is simulated by augmenting a pre-generated, physical environment-aligned 3D reality mesh. A camera system captures digital images of a site. A backend processing system generates a 3D reality mesh of the site based on the digital images, assigns metadata to infrastructure elements in the 3D reality mesh, and stores the 3D reality mesh and metadata. At a subsequent time, a mobile device accesses the 3D reality mesh and assigned metadata. A positioning sensor determines a live position of the mobile device. An orientation sensor determines a live orientation of the mobile device. Based on the live position and live orientation, a view of the 3D reality mesh aligned to the physical environment is produced and displayed on a display device of the mobile device. That view is then augmented based on the assigned metadata.

Claims (72)

1. A method for simulating augmented reality by augmenting a pre-generated, physical environment-aligned three dimensional (3D) reality mesh, comprising:

capturing, by a camera system, digital images of a site;

generating a 3D reality mesh of the site based on the digital images;

assigning metadata to infrastructure elements in the 3D reality mesh;

at a time subsequent to the generating and assigning, accessing the 3D reality mesh and assigned metadata on a mobile device;

determining, by a position sensor, a live position of the mobile device;

determining, by an orientation sensor, a live orientation of the mobile device;

based on the live position and live orientation, displaying on a display device of the mobile device, a view of the 3D reality mesh aligned to the physical environment; and

augmenting the displayed view of the physical environment-aligned 3D reality mesh based on the assigned metadata.

2. The method of claim 1 , further comprising:

generating configuration information describing how metadata should be displayed via augmentations; and

at the time subsequent to the generating and assigning, accessing the configuration information on the mobile device.

3. The method of claim 1 , further comprising:

adjusting the live position based on a user supplied position to correct errors in position determined by the positioning sensor.

4. The method of claim 1 , further comprising:

adjusting the live orientation based on a user supplied orientation to correct errors in orientation determined by the orientation sensor.

5. The method of claim 1 , further comprising

decoupling the live position and live orientation from an actual position and orientation of the mobile device, to generate a third person view.

6. The method of claim 1 , wherein the augmenting further comprises:

highlighting infrastructure elements in the 3D reality mesh to indicate that augmentations with data are available for the infrastructure elements upon selection.

7. The method of claim 1 , wherein the augmenting further comprises:

imposing a 3D window with data nearby a selected infrastructure element in the 3D reality mesh.

8. The method of claim 1 , wherein the augmenting further comprises:

projecting a two dimensional (2D) map upon a surface in the 3D reality mesh.

9. The method of claim 1 , wherein the generating and the assigning are performed by a backend processing system that stores the 3D reality mesh and assigned metadata until accessed by the mobile device.

10. The method of claim 1 , wherein the mobile device is a tablet computer.

11. The method of claim 1 , further comprising:

capturing, by a laser scanner, point clouds of the site,

wherein the generating the 3D reality mesh of the site is further based on the point clouds.

12. A system for simulating augmented reality by augmenting a pre-generated, physical environment-aligned three dimensional (3D) reality mesh, comprising:

a camera system configured to capture digital images of a site;

a backend processing system configured to receive the digital images and execute a reality modeling application that generates a 3D reality mesh of the site based on the digital images, and execute a data processing application that assigns metadata to infrastructure elements in the 3D reality mesh; and

a mobile device configured to access the 3D reality mesh and assigned metadata, and execute a viewer application that, based on a live position and live orientation of the mobile device, displays a view of the 3D reality mesh aligned to the physical environment, and augments the displayed view of the physical environment-aligned 3D reality mesh based on the assigned metadata.

13. The system of claim 12 , wherein the data processing application further generates configuration information describing how metadata should be displayed via augmentations.

14. The system of claim 12 , wherein the viewer application further adjusts the live position based on a user supplied position to correct errors in position.

15. The system of claim 12 , wherein the viewer application further adjusts the live orientation based on a user supplied orientation to correct errors in orientation.

16. The system of claim 12 , wherein the mobile device is a tablet computer.

17. A non-transitory electronic device-readable medium that includes instructions executable on one or more electronic-devices that, when executed, are operable to:

at a first time,

generate a three-dimensional (3D) reality mesh of a site,

assign metadata to infrastructure elements in the 3D reality mesh,

store the 3D reality mesh and the metadata until a subsequent time; and

at the subsequent time,

determine a live position,

determine a live orientation,

based on the live position and live orientation, display a view of the 3D reality mesh aligned to the physical environment, and

augment the displayed view of the physical environment-aligned 3D reality mesh based on the assigned metadata.

18. The non-transitory electronic device-readable medium of claim 17 , wherein the instructions, when executed, are further operable to:

at the subsequent time, adjust the live position based on a user supplied position to correct errors in position.

19. The non-transitory electronic device-readable medium of claim 17 , wherein the instructions, when executed, are further operable to:

adjust the live orientation based on a user supplied orientation to correct errors in orientation.

20. The non-transitory electronic device-readable medium of claim 17 , wherein the instructions, when executed, are further operable to:

decouple the live position and live orientation from an actual position and orientation to generate a third person view.

21. A method for simulating augmented reality by augmenting a pre-generated, physical environment-aligned mesh, comprising:

accessing, on a mobile device, a pre-generated mesh that represents a site having one or more infrastructure elements, and metadata assigned to the infrastructure elements;

determining a live position of the mobile device;

determining a live orientation of the mobile device;

based on the live position and live orientation, displaying on a display device of the mobile device, a view of the mesh aligned to the physical environment; and

augmenting the displayed view of the physical environment-aligned mesh based on the assigned metadata.

22. The method of claim 21 , further comprising:

capturing, by a camera system, digital images of the site;

generating the mesh that represents the site based on the digital images; and

assigning metadata to infrastructure elements in the mesh.

23. The method of claim 21 , further comprising:

generating configuration information describing how metadata should be displayed via augmentations; and

accessing the configuration information on the mobile device.

24. The method of claim 21 , wherein the determining the live position is performed by a positioning sensor of the mobile device, and the method further comprises:

adjusting the live position based on a user supplied position to correct errors in position determined by the positioning sensor.

25. The method of claim 21 , wherein the determining the live orientation is performed by an orientation sensor of the mobile device, and the method further comprises:

adjusting the live orientation based on a user supplied orientation to correct errors in orientation determined by the orientation sensor.

26. The method of claim 21 , further comprising

decoupling the live position and live orientation from an actual position and orientation of the mobile device, to generate a third person view.

Assignments (3)
SECURITY INTEREST Recorded Oct 25, 2024
From: BENTLEY SYSTEMS, INCORPORATED
To: PNC BANK, NATIONAL ASSOCIATION
Reel/Frame 069268/0042 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Feb 23, 2018
From: CÔTÉ, STÉPHANE; MÉNARD, MAXIME
To: BENTLEY SYSTEMS, INCORPORATED
Reel/Frame 045022/0163 →
SECURITY INTEREST Recorded Dec 21, 2017
From: BENTLEY SYSTEMS INCORPORATED
To: PNC BANK, NATIONAL ASSOCIATION
Reel/Frame 044938/0083 →
Cited By (1)
US 12,340,470