Granted Patent B2

US 9,865,058 · App. 14/184,294 · Granted Jan 9, 2018

Inventor: Brian Mullins (Sierra Madre, CA)

Assignee: DAQRI, LLC

G06T7/004G06T19/006

Abstract

Claims (56)

1. A server comprising:

a storage device storing instructions; and

a hardware processor communicatively coupled to the storage device and configured by the instructions to:

receive video feeds, location information, and orientation information from a plurality of wearable devices;

map an object and construct a three-dimensional model of the object using two-dimensional images captured from different angles and locations around the mapped object from the plurality of wearable devices, the three-dimensional model being constructed based on the video feeds, the location information, and the orientation information received from the plurality of wearable devices;

perform analytics from the video feeds to identify a manipulation on the object, to update the three-dimensional model of the object based on the manipulation on the object, and to generate a dynamic status of the object based on the manipulation on the object with respect to reference data related to the object, the dynamic status including a completion rate and a compliance rate, and the reference data comprising another three-dimensional model of a reference object, a reference process for completing the reference object, and a reference compliance related to the reference process;

store the reference data and the dynamic status identifying the wearable device associated with the manipulation on the object; and

detect and identify a non-compliance based on the manipulation on the object in relation to the reference compliance by: identifying a non-compliance of a first manipulation on the object, identifying a first wearable device corresponding to the first manipulation on the object, identifying a second wearable device associated with a second manipulation dependent on the first manipulation, and notifying the second wearable device of the non-compliance of the first manipulation on the object from the first wearable device.

2. The server of claim 1 , wherein the manipulation of the object comprises a modification of an existing component on the object, an addition of a new component to the object, or a removal of an existing component on the object.

3. The server of claim 1 , wherein the hardware processor is further configured by the instructions to perform object recognition on the video feeds actively or against the reference data to identify a component on the object or within a scene or a periphery from a wearable device.

4. The server of claim 1 , wherein the dynamic status comprising an identification of a type of manipulation on the object, an identification of a tool used in the manipulation on the object, a location of the manipulation relative to the three-dimensional model of the object, and an identification of the wearable device associated with the manipulation on the object.

5. The server of claim 1 , wherein the hardware processor is further configured by the instructions to:

update the dynamic status related to the manipulation on the object, the dynamic status identifying a completion status of the object with respect to the reference object;

keep an inventory of components on the object;

generate a history of manipulations of the components and corresponding wearable devices; and

enable communication between two or more wearable devices determined based on a corresponding manipulation of the object.

6. The server of claim 5 , wherein to enable communication between two or more wearable devices determined based on the corresponding manipulation of the object, the hardware processor is configured by the instructions to identify the first wearable device corresponding to a first manipulation on the object, to identify a second wearable device associated with a second manipulation similar to the first manipulation, and to enable communication between the second wearable device and the first wearable device in response to receiving a request for assistance on the first manipulation on the object from the first wearable device.

7. The server of claim 1 , wherein the hardware processor is further configured by the instructions to receive, from a client, a request for a location of a component related to the manipulation on the object, to identify and communicate the location of the component relative to the reference data to the client.

8. The server of claim 1 , wherein the processor comprises an offload application configured to:

receive a request from the plurality of wearable devices to offload at least one of a tracking process and an augmented reality rendering process, the augmented reality rendering process based on an augmented reality database;

generate offloaded processed data based on the request and the location and the orientation of the wearable device; and

stream the offloaded processed data to the plurality of wearable devices in response to the request, the plurality of wearable devices configured to generate a visualization of the offloaded processed data in a corresponding display of the plurality of wearable devices.

9. A method comprising:

receiving video feeds, location information, and orientation information from a plurality of wearable devices;

mapping an object;

constructing a three-dimensional model of the object using two-dimensional images captured from different angles and locations around the mapped object from the plurality of wearable devices, the three-dimensional model being constructed based on the video feeds, location information, and orientation information received from the plurality of wearable devices;

performing analytics from the video feeds to identify a manipulation on the object, to update the three-dimensional model of the object based on the manipulation on the object, and to generate a dynamic status of the object based on the manipulation on the object with respect to reference data related to the object, the dynamic status including a completion rate and a compliance rate;

generating a survey of components associated with the three-dimensional model of the object;

storing the reference data and the dynamic status identifying the wearable device associated with the manipulation on the object in a storage device; and

enabling communication between two or more wearable devices determined based on a corresponding manipulation of the object by: identifying a first wearable device corresponding to a first manipulation on the object, identifying a second wearable device associated with a second manipulation similar to the first manipulation, and enabling communication between the second wearable device and the first wearable device in response to receiving a request for assistance on the first manipulation on the object from the first wearable device.

10. The method of claim 9 , wherein the manipulation of the object comprises a modification of an existing component on the object, an addition of a new component to the object, or a removal of an existing component on the object.

11. The method of claim 9 , further comprising:

performing object recognition on the video feeds actively or against the reference data to identify a component on the object or within a scene or a periphery from a wearable device.

12. The method of claim 9 , wherein the dynamic status comprises an identification of a type of manipulation on the object, an identification of a tool used in the manipulation on the object, a location of the manipulation relative to the three-dimensional model of the object, and an identification of the wearable device associated with the manipulation on the object.

13. The method of claim 9 , wherein the reference data comprises a three-dimensional model of a reference object, a reference process for completing the reference object, and a reference compliance related to the reference process.

14. The method of claim 13 , further comprising:

updating the dynamic status related to the manipulation on the object, the dynamic status identifying a completion status of the object with respect to the reference object;

detecting and identifying a non-compliance based on the manipulation on the object in relation to the reference compliance;

keeping an inventory of components on the object; and

generating a history of manipulations of the components and corresponding wearable devices.

15. The method of claim 14 , further comprising:

identifying a non-compliance of a first manipulation on the object;

identifying the first wearable device corresponding to the first manipulation on the object;

identifying a second wearable device associated with a second manipulation dependent on the first manipulation; and

notifying the second wearable device of the non-compliance of the first manipulation on the object from the first wearable device.

16. The method of claim 9 , further comprising:

receiving, from a client, a request for a location of a component related to the manipulation on the object; and

identifying and communicating the location of the component relative to the reference data to the client.

17. A non-transitory machine-readable medium comprising instructions that, when executed by one or more processors of a machine, cause the machine to perform operations comprising:

receiving video feeds, location information, and orientation information from a plurality of wearable devices;

mapping an object;

constructing a three-dimensional model of the object using two-dimensional images captured from different angles and locations around the mapped object, the three-dimensional model being constructed based on the video feeds, location information, and orientation information received from the plurality of wearable devices;

performing analytics from the video feeds to identify a manipulation on the object, to update the three-dimensional model of the object based on the manipulation on the object, and to generate a dynamic status of the object based on the manipulation on the object with respect to reference data related to the object, the dynamic status including a completion rate and a compliance rate;

generating a survey of components associated with the three-dimensional model of the object;

storing the reference data and the dynamic status identifying the wearable device associated with the manipulation on the object in a storage device; and

enabling communication between two or more wearable devices determined based on a corresponding manipulation of the object by: identifying a first wearable device corresponding to a first manipulation on the object, identifying a second wearable device associated with a second manipulation similar to the first manipulation, and enabling communication between the second wearable device and the first wearable device in response to receiving a request for assistance on the first manipulation on the object from the first wearable device.

Assignments (12)

Continuity (1)