IP Library Granted Patent US 12,333,657
Granted Patent B2
US 12,333,657 · App. 18/072,525 · Granted Jun 17, 2025

Building data platform with augmented reality based digital twins

Inventors: Erik S. Paulson (Madison, WI); Brian S. Otto (Menomonee Falls, WI); Todd Leister (Milwaukee, WI); Aureliu Maxim (Milwaukee, WI)
Assignee: TYCO FIRE & SECURITY GMBH
G06T19/006
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,333,657
App. No.
18/072,525
Granted
Jun 17, 2025
Kind
B2
Abstract

A system can operate to receive an indication of a point in a space of an environment, wherein an augmented reality system is located in the space of the environment at a coordinate in a first coordinate system. The system can operate to query a digital twin for a coordinate of the point in a second coordinate system, the digital twin including a contextual description of the environment. The system can operate to translate the coordinate of the augmented reality system in the first coordinate system to a coordinate of the augmented reality system in the second coordinate system based on the coordinate of the point in the second coordinate system and the coordinate of the augmented reality system in the first coordinate system. The system can operate to generate data to cause the augmented reality system to display information based at least in part on the translation.

Claims (98)

1. A system, comprising:

one or more memory devices storing instructions thereon that, when executed by one or more processors, cause the one or more processors to:

receive an indication of a point in a space of an environment, wherein an augmented reality system is located in the space of the environment at a coordinate in a first coordinate system;

query, using the indication of the point, a digital twin to identify a coordinate of the point in a second coordinate system, the querying identifying a representation of the point stored in the digital twin and identifying a relationship of the digital twin connected to the identified representation of the point and extending between the coordinate of the point in the second coordinate system and the identified representation of the point stored in the digital twin, the digital twin comprising a contextual description of the environment;

translate the coordinate of the augmented reality system in the first coordinate system to a coordinate of the augmented reality system in the second coordinate system based on the coordinate of the point in the second coordinate system and the coordinate of the augmented reality system in the first coordinate system; and

generate data to cause the augmented reality system to display information based at least in part on the translation.

2. The system of claim 1 , wherein the point is at least one of:

a quick response (QR) code;

a piece of art;

a piece of equipment; or

a feature of a building.

3. The system of claim 1 , wherein the instructions cause the one or more processors to:

receive, from the augmented reality system, a coordinate of the point in the first coordinate system; and

translate the coordinate of the augmented reality system in the first coordinate system to the coordinate of the augmented reality system in the second coordinate system based on:

the coordinate of the point in the first coordinate system;

the coordinate of the point in the second coordinate system; and

the coordinate of the augmented reality system in the first coordinate system.

4. The system of claim 1 , wherein:

the environment is a building and the space is a space of the building;

the point is a building object located within the space of the building; and

the information is a description of the space of the building.

5. The system of claim 1 , wherein the instructions cause the one or more processors to:

query the digital twin for the information, the information describing the space;

generate the data based on a query response of the digital twin; and

transmit the data to the augmented reality system to cause the augmented reality system to display a view of the space augmented with the information.

6. The system of claim 1 , comprising:

the augmented reality system configured to:

receive the data from the one or more processors; and

display a view of the space augmented with the information.

7. The system of claim 1 , wherein the instructions cause the one or more processors to:

receive, from the augmented reality system, an indication of an asset located within the environment;

retrieve, based on the indication of the asset and the translation, a description of the asset from the digital twin; and

generate the data to cause the augmented reality system to display the description of the asset.

8. The system of claim 1 , wherein the instructions cause the one or more processors to:

receive, from the augmented reality system, a coordinate of an asset within the environment in the first coordinate system;

translate the coordinate of the asset in the first coordinate system to the second coordinate system; and

identify the asset within the digital twin based on a comparison of the coordinate of the asset in the second coordinate system and a stored coordinate of the asset in the second coordinate system stored within the digital twin.

9. The system of claim 1 , wherein the instructions cause the one or more processors to:

receive an account profile of a user of the augmented reality system;

determine, based on the account profile, that the user associated with the account profile has access to the information or authorization to utilize the point for the translation; and

generate data to cause the augmented reality system to display the information responsive to a determination that the user has access to the information or authorization to utilize the point for the translation.

10. The system of claim 1 , wherein the instructions cause the one or more processors to:

receive a set of coordinates in the first coordinate system of the augmented reality system; and

translate the set of coordinates of the augmented reality system in the first coordinate system to a second set of coordinates in the second coordinate system based on the coordinate of the point in the second coordinate system and the set of coordinates of the augmented reality system in the first coordinate system.

11. The system of claim 1 , wherein the instructions cause the one or more processors to:

generate data to cause a user device to display a graphical representation of the space of the environment;

receive, from the user device, a selection of the point in the graphical representation of the space of the environment;

identify, based on the selection of the point in the graphical representation of the space of the environment, the coordinate of the point in the second coordinate system; and

save the coordinate of the point in the second coordinate system in the digital twin.

12. The system of claim 1 , comprising:

the digital twin comprising a graph data structure comprising a plurality of nodes representing entities of the environment and a plurality of edges between the plurality of nodes, the plurality of edges representing relationships between the entities;

wherein the instructions cause the one or more processors to:

receive the coordinate of the point in the second coordinate system;

identify a first node of the plurality of nodes representing the point;

generate or update a second node of the plurality of nodes related to the first node by an edge of the plurality of edges to cause the second node to store or link to the coordinate of the point in the second coordinate system.

13. The system of claim 1 , comprising:

the digital twin comprising a graph data structure comprising a plurality of nodes representing entities of the environment and a plurality of edges between the plurality of nodes, the plurality of edges representing relationships between the entities;

wherein the instructions cause the one or more processors to query the digital twin for the coordinate of the point in the second coordinate system by:

identifying a first node of the plurality of nodes representing the point;

identifying an edge of the plurality of edges relating the first node to a second node; and

retrieving the coordinate of the point in the second coordinate system based on the second node.

14. The system of claim 1 , comprising:

the digital twin comprising a graph data structure comprising a plurality of nodes representing entities of the environment and a plurality of edges between the plurality of nodes, the plurality of edges representing relationships between the entities;

wherein the information describes an entity of the environment;

wherein the instructions cause the one or more processors to:

query the digital twin for the information by:

identifying a first node of the plurality of nodes representing the entity;

identifying an edge of the plurality of edges relating the first node to a second node representing a coordinate of the entity in the second coordinate system;

determining that the coordinate of the entity in the second coordinate system indicates that the entity is within a distance from the augmented reality system; and

retrieving the information of the entity responsive to a determination that the entity is within the distance from the augmented reality system.

15. A method, comprising:

receiving, by one or more processing circuits, an indication of a point in a space of an environment, wherein an augmented reality system is located in the space of the environment at a coordinate in a first coordinate system;

querying, by the one or more processing circuits, using the indication of the point, a digital twin to identify a coordinate of the point in a second coordinate system, the querying identifying a representation of the point stored in the digital twin and identifying a relationship of the digital twin connected to the identified representation of the point and extending between the coordinate of the point in the second coordinate system and the identified representation of the point stored in the digital twin, the digital twin comprising a contextual description of the environment;

translating, by the one or more processing circuits, the coordinate of the augmented reality system in the first coordinate system to a coordinate of the augmented reality system in the second coordinate system based on the coordinate of the point in the second coordinate system and the coordinate of the augmented reality system in the first coordinate system; and

generating, by the one or more processing circuits, data to cause the augmented reality system to display information based at least in part on the translation.

16. The method of claim 15 , comprising:

receiving, from the augmented reality system, a coordinate of the point in the first coordinate system; and

translating, by the one or more processing circuits, the coordinate of the augmented reality system in the first coordinate system to the coordinate of the augmented reality system in the second coordinate system based on:

the coordinate of the point in the first coordinate system;

the coordinate of the point in the second coordinate system; and

the coordinate of the augmented reality system in the first coordinate system.

17. The method of claim 15 , comprising:

querying, by the one or more processing circuits, the digital twin for the information, the information describing the space;

generating, by the one or more processing circuits, the data based on a query response of the digital twin; and

transmitting, by the one or more processing circuits, the data to the augmented reality system to cause the augmented reality system to display a view of the space augmented with the information.

18. The method of claim 15 , comprising:

receiving, by the one or more processing circuits from the augmented reality system, an indication of an asset located within the environment;

retrieving, by the one or more processing circuits, based on the indication of the asset and the translation, a description of the asset from the digital twin; and

generating, by the one or more processing circuits, the data to cause the augmented reality system to display the description of the asset.

19. The method of claim 15 , comprising:

receiving, by the one or more processing circuits, a set of coordinates in the first coordinate system of the augmented reality system; and

translating, by the one or more processing circuits, the set of coordinates of the augmented reality system in the first coordinate system to a second set of coordinates in the second coordinate system based on the coordinate of the point in the second coordinate system and the set of coordinates of the augmented reality system in the first coordinate system.

20. A building system, comprising:

one or more processors configured to execute instructions stored on one or more memory devices, the instructions causing the one or more processors to:

receive an indication of a point in a space of a building, wherein an augmented reality system is located in the space of the building at a coordinate in a first coordinate system;

query, using the indication of the point, a digital twin to identify a coordinate of the point in a second coordinate system, the querying identifying a representation of the point stored in the digital twin and identifying a relationship of the digital twin connected to the identified representation of the point and extending between the coordinate of the point in the second coordinate system and the identified representation of the point stored in the digital twin, the digital twin comprising a contextual description of the building;

translate the coordinate of the augmented reality system in the first coordinate system to a coordinate of the augmented reality system in the second coordinate system based on the coordinate of the point in the second coordinate system and the coordinate of the augmented reality system in the first coordinate system; and

generate data to cause the augmented reality system to display information based at least in part on the translation.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Apr 9, 2024
From: JOHNSON CONTROLS TYCO IP HOLDINGS LLP
To: TYCO FIRE & SECURITY GMBH
Reel/Frame 067056/0552 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 12, 2022
From: PAULSON, ERIK S.; OTTO, BRIAN S.; LEISTER, TODD; MAXIM, AURELIU
To: JOHNSON CONTROLS TYCO IP HOLDINGS LLP
Reel/Frame 062052/0075 →