IP Library Granted Patent US 11,341,652
Granted Patent B2
US 11,341,652 · App. 16/923,010 · Granted May 24, 2022

Three-dimensional mapping system

Inventor: Brian Mullins (Altadena, CA)
Assignee: FACEBOOK TECHNOLOGIES, LLC
G06T7/149G06T7/70G06T19/006
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,341,652
App. No.
16/923,010
Granted
May 24, 2022
Kind
B2
Abstract

A survey application generates a survey of components associated with a three-dimensional model of an object. The survey application receives video feeds, location information, and orientation information from wearable devices in proximity to the object. The three-dimensional model of the object is generated based on the video feeds, sensor data, location information, and orientation information received from the wearable devices. Analytics is performed from the video feeds to identify a manipulation on the object. The three-dimensional model of the object is updated based on the manipulation on the object. A dynamic status related to the manipulation on the object is generated with respect to reference data related the object. A survey of components associated with the three-dimensional model of the object is generated.

Claims (63)

1. A server comprising:

a storage device storing instructions; and

a hardware processor communicatively coupled to the storage device and configured to perform operations based on the instructions, the operations comprising:

receiving video feeds, location information, and orientation information from a plurality of wearable devices;

dynamically generating a three-dimensional model of a physical object based on the video feeds, the location information, and the orientation information received from the plurality of wearable devices; and

performing analytics from the video feeds to identify a manipulation on the physical object and to generate a dynamic status of the physical object based on the manipulation on the physical object with respect to reference data related to the physical object, the reference data comprising a reference object, and the dynamic status identifying a completion status of the physical object based on a comparison of the physical object to the reference object;

receiving a request from the plurality of wearable devices to offload at least one of a tracking process or an augmented reality rendering process, the augmented reality rendering process being based on an augmented reality database;

generating offloaded processed data based on the request, the location, and the orientation of the wearable device; and

streaming the offloaded processed data to the plurality of wearable devices in response to the request, the plurality of wearable devices being configured to generate a visualization of the offloaded processed data in a corresponding display of the plurality of wearable devices.

2. The server of claim 1 , wherein the operations further comprise:

receiving, from a client, a request for a dynamic status of the physical object; and

in response to the request, sending, to the client, the dynamic status of the physical object.

3. The server of claim 1 , wherein the manipulation of the physical object comprises a modification of at least one of an existing component on the physical object, an addition of a new component to the physical object, or a removal of an existing component on the physical object.

4. The server of claim 1 , wherein the operations further comprise:

performing object recognition on the video feeds to identify a particular component on the physical object, within a scene or within a periphery from a wearable device.

5. The server of claim 1 , wherein the dynamic status comprises at least one of an identification of a type of manipulation on the physical object, an identification of a tool used in the manipulation on the physical object, a location of the manipulation relative to the three-dimensional model of the physical object, or an identification of the wearable device associated with the manipulation on the physical object.

6. The server of claim 1 , wherein the operations further comprise:

updating the dynamic status related to the manipulation on the physical object.

7. The server of claim 1 , wherein the operations further comprise:

enabling communication between two or more wearable devices, the two or more wearable devices identified based on the manipulation of the physical object.

8. The server of claim 7 , wherein the enabling communication between the two or more wearable devices comprises:

identifying a non-compliance of a first manipulation on the physical object;

identifying a first wearable device corresponding to the first manipulation on the physical object;

identifying a second wearable device associated with a second manipulation dependent on the first manipulation; and

notifying the second wearable device of the non-compliance of the first manipulation on the physical object.

9. The server of claim 7 , wherein the enabling communication between the two or more wearable devices comprises:

identifying a first wearable device corresponding to a first manipulation on the physical object;

identifying a second wearable device associated with a second manipulation similar to the first manipulation;

receiving, from the first wearable device, a request for assistance on the first manipulation on the physical object; and

enabling communication between the second wearable device and the first wearable device in response to the request.

10. A method comprising:

receiving video feeds, location information, and orientation information from a plurality of wearable devices;

dynamically generating a three-dimensional model of a physical object based on the video feeds, the location information, and the orientation information received from the plurality of wearable devices; and

performing analytics from the video feeds to identify a manipulation on the physical object and to generate a dynamic status of the physical object based on the manipulation on the physical object with respect to reference data related to the physical object, the reference data comprising a reference object, and the dynamic status identifying a completion status of the physical object based on a comparison of the physical object to the reference object;

receiving a request from the plurality of wearable devices to offload at least one of a tracking process or an augmented reality rendering process, the augmented reality rendering process being based on an augmented reality database;

generating offloaded processed data based on the request, the location, and the orientation of the wearable device; and

streaming the offloaded processed data to the plurality of wearable devices in response to the request, the plurality of wearable devices being configured to generate a visualization of the offloaded processed data in a corresponding display of the plurality of wearable devices.

11. The method of claim 10 , further comprising:

receiving, from a client, a request for a dynamic status of the physical object; and

in response to the request, sending, to the client, the dynamic status of the physical object.

12. The method of claim 10 , wherein the manipulation of the physical object comprises a modification of at least one of an existing component on the physical object, an addition of a new component to the physical object, or a removal of an existing component on the physical object.

13. The method of claim 10 , further comprising:

performing object recognition on the video feeds to identify a particular component on the physical object, within a scene or within a periphery from a wearable device.

14. The method of claim 10 , wherein the dynamic status comprises at least one of an identification of a type of manipulation on the physical object, an identification of a tool used in the manipulation on the physical object, a location of the manipulation relative to the three-dimensional model of the physical object, or an identification of the wearable device associated with the manipulation on the physical object.

15. The method of claim 10 , further comprising:

updating the dynamic status related to the manipulation on the physical object.

16. The method of claim 10 , further comprising:

enabling communication between two or more wearable devices, the two or more wearable devices identified based on the manipulation of the physical object.

17. The method of claim 16 , wherein the enabling communication between the two or more wearable devices comprises:

identifying a first wearable device corresponding to a first manipulation on the physical object; identifying a second wearable device associated with a second manipulation similar to the first manipulation;

receiving, from the first wearable device, a request for assistance on the first manipulation on the physical object; and

enabling communication between the second wearable device and the first wearable device in response to the request.

18. The method of claim 10 , wherein the enabling communication between the two or more wearable devices comprises:

identifying a non-compliance of a first manipulation on the physical object;

identifying a first wearable device corresponding to the first manipulation on the physical object;

identifying a second wearable device associated with a second manipulation dependent on the first manipulation; and

notifying the second wearable device of the non-compliance of the first manipulation on the physical object.

19. A non-transitory machine-readable medium comprising instructions that, when executed by one or more processors of a machine, cause the machine to perform operations comprising:

receiving video feeds, location information, and orientation information from a plurality of wearable devices;

dynamically generating a three-dimensional model of a physical object based on the video feeds, the location information, and the orientation information received from the plurality of wearable devices; and performing analytics from the video feeds to identify a manipulation on the physical object and to generate a dynamic status of the physical object based on the manipulation on the physical object with respect to reference data related to the physical object, the reference data comprising a reference object, and the dynamic status identifying a completion status of the physical object based on a comparison of the physical object to the reference object;

receiving a request from the plurality of wearable devices to offload at least one of a tracking process or an augmented reality rendering process, the augmented reality rendering process being based on an augmented reality database;

generating offloaded processed data based on the request, the location, and the orientation of the wearable device; and

streaming the offloaded processed data to the plurality of wearable devices in response to the request, the plurality of wearable devices being configured to generate a visualization of the offloaded processed data in a corresponding display of the plurality of wearable devices.

Assignments (2)
CHANGE OF NAME Recorded Aug 3, 2022
From: FACEBOOK TECHNOLOGIES, LLC
To: META PLATFORMS TECHNOLOGIES, LLC
Reel/Frame 060936/0494 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 2, 2021
From: RPX CORPORATION
To: FACEBOOK TECHNOLOGIES, LLC
Reel/Frame 056777/0588 →
Continuity (3)
Continuation 15711317 · Sep 21, 2017
Continuation 14184294 · Feb 19, 2014
Related Publication 20200349715A1 · Nov 5, 2020