Granted Patent B2

US 12,333,759 · App. 18/246,464 · Granted Jun 17, 2025

Inventors: Shengzhi Wu (Shenzhen, CN); Alexander James Faaborg (Mountain View, CA)

Assignee: Google LLC

G06T7/74G06F3/04815G06F3/0484G06T15/00G06V20/20G06T2200/24

Abstract

Claims (59)

1. A method comprising:

receiving image data from an image sensor on a user device;

generating first identification data about a first controllable device included in the image data;

determining that a first three-dimensional (3D) map from a map database is annotated with second identification data that corresponds to the first identification data, the map database storing a plurality of 3D maps including the first 3D map and a second 3D map, the first 3D map being associated with the first controllable device, the first 3D map being generated during a configuration process of the first controllable device, the second 3D map being associated with a second controllable device;

comparing the image data to the first 3D map to determine that the first 3D map at least partially corresponds to a physical space represented by the image data;

obtaining a 3D position of the first controllable device in the physical space from the first 3D map; and

rendering a user interface object on a display of the user device in a location that is based on the 3D position of the first controllable device, the user interface object including a control for interacting with the first controllable device.

2. The method of claim 1 , wherein the 3D position includes a six degree of freedom position of the first controllable device.

3. The method of claim 1 , wherein the first identification data includes information about at least one of a device name or a device type of the first controllable device.

4. The method of claim 1 , further comprising:

generating a plurality of feature points about the physical space that includes the first controllable device;

generating the first 3D map based on the plurality of feature points, the first 3D map identifying the 3D position of the first controllable device in the physical space; and

storing the first 3D map in the map database.

5. The method of claim 1 , further comprising:

receiving a selection to the control on the user interface object; and

initiating execution of a function on the first controllable device by communicating with the first controllable device.

6. A method comprising:

receiving image data from an image sensor on a user device;

generating identification data about a first controllable device based on the image data, the identification data including information about a classification of a physical space that includes the first controllable device;

identifying, using the identification data, a first three-dimensional (3D) map from a map database that stores a plurality of 3D maps including the first 3D map and a second 3D map, the first 3D map being associated with the first controllable device, the first 3D map being generated during a configuration process of the first controllable device, the second 3D map being associated with a second controllable device;

comparing the image data to the first 3D map to determine that the first 3D map at least partially corresponds to the physical space represented by the image data;

obtaining a 3D position of the first controllable device in the physical space from the first 3D map; and

rendering a user interface object on a display of the user device in a location that is based on the 3D position of the first controllable device, the user interface object including a control for interacting with the first controllable device.

7. A method comprising:

receiving image data from an image sensor on a first user device;

generating identification data about a first controllable device based on the image data;

identifying, using the identification data, a first three-dimensional (3D) map from a map database that stores a plurality of 3D maps including the first 3D map and a second 3D map, the first 3D map being associated with the first controllable device, the second 3D map being associated with a second controllable device;

comparing the image data to the first 3D map to determine that the first 3D map at least partially corresponds to a physical space represented by the image data;

obtaining a 3D position of the first controllable device in the physical space from the first 3D map;

rendering a user interface object on a display of the first user device in a location that is based on the 3D position of the first controllable device, the user interface object including a control for interacting with the first controllable device; and

transmitting an identifier of the first 3D map to a second user device.

8. A non-transitory computer-readable medium storing executable instructions that when executed by at least one processor are configured to cause the at least one processor to execute operations, the operations comprising:

receiving image data from an image sensor on a user device;

generating first identification data about a first controllable device included in the image data;

determining that a first three-dimensional (3D) map from a map database is annotated with second identification data that corresponds to the first identification data, the map database storing a plurality of 3D maps including the first 3D map and a second 3D map, the first 3D map being associated with the first controllable device, the second 3D map being associated with a second controllable device, the first 3D map being generated during a configuration process of the first controllable device, the first and second 3D maps representing different physical spaces;

comparing the image data to the first 3D map to determine that the first 3D map at least partially corresponds to a physical space represented by the image data;

obtaining a 3D position of the first controllable device in the physical space from the first 3D map; and

rendering a user interface object on a display of the user device in a location that is based on the 3D position of the first controllable device, the user interface object including a control for interacting with the first controllable device.

9. The non-transitory computer-readable medium of claim 8 , wherein the user interface object displays information about an action of the first controllable device.

10. The non-transitory computer-readable medium of claim 8 , wherein the first identification data includes information about at least one of a device name, a device type, or a space type of the physical space that includes the first controllable device.

11. The non-transitory computer-readable medium of claim 8 , wherein the user device is a first user device, wherein the operations further comprise:

transmitting an identifier of the first 3D map to a second user device, the second user device configured to use the identifier to join an augment reality (AR) session that includes at least a portion of the physical space.

12. A computing system comprising:

at least one processor; and

a non-transitory computer-readable medium storing executable instructions that cause the at least one processor to:

receive image data from an image sensor;

generate first identification data about a first controllable device included in the image data;

determine that a first three-dimensional (3D) map from a map database is annotated with second identification data that corresponds to the first identification data, the map database storing a plurality of 3D maps including the first 3D map and a second 3D map, the first 3D map being associated with the first controllable device, the second 3D map being associated with a second controllable device, the first 3D map being generated during a configuration process of the first controllable device, the first and second 3D maps representing different physical spaces;

compare the image data to the first 3D map to determine that the first 3D map at least partially corresponds to a physical space represented by the image data;

obtain a 3D position of the first controllable device in the physical space from the first 3D map; and

render a user interface object on a display in a location that is based on the 3D position of the first controllable device, the user interface object including a control for interacting with the first controllable device.

13. The computing system of claim 12 , wherein the first identification data includes information about a device name and a device type of the first controllable device.

14. The computing system of claim 12 , wherein the executable instructions include instructions that cause the at least one processor to:

generate, during the configuration process of the first controllable device, a plurality of feature points about the physical space that includes the first controllable device;

generate the first 3D map based on the plurality of feature points, the first 3D map identifying the 3D position of the first controllable device in the physical space; and

store the first 3D map in the map database.

15. The computing system of claim 12 , wherein the executable instructions include instructions that cause the at least one processor to:

transmit an identifier of the first 3D map to a user device, the user device configured to use the identifier to join an extended reality session that includes at least a portion of the physical space.

16. The computing system of claim 12 , wherein the computing system includes a head-mounted display device.

Assignments (1)

Continuity (1)

References Cited (48)

Cited By (1)

US 12,724,866