Granted Patent B2

US 9,824,437 · App. 14/966,666 · Granted Nov 21, 2017

Inventor: Brian Mullins (Sierra Madre, CA)

Assignee: DAQRI, LLC

G06T7/0008G02B27/0093G02B27/0172G06T19/006G02B2027/014G02B2027/0138G06T2207/30164G06T2219/024

Abstract

Claims (67)

1. A method comprising:

receiving, at a server, sensor data from a plurality of mobile devices having optical sensors and location sensors, the sensor data identifying a mobile physical object and a location of the mobile physical object within a predefined geographic region based on the optical and location sensors;

identifying mobile physical objects that are present and absent within the predefined geographic region based on a predefined catalog of the mobile physical objects at the predefined geographic region;

generating, at the server, a real-time inventory of the mobile physical objects at the predefined geographic region based on the mobile physical objects that are present and absent within the predefined geographic region, the real-time inventory including the identification of the mobile physical objects and the location of the mobile physical objects within the geographic region;

receiving datasets from the plurality of mobile devices, each dataset identifying a task selected in an augmented reality application of a corresponding mobile device, an identification of a physical tool detected at the corresponding mobile device, each mobile device including a display device;

identifying physical tools present and absent at a dedicated tool board, the dedicated tool board configured to store the physical tools for the tasks of the augmented reality application;

comparing an identification of the physical tools present and absent at the dedicated tool board with the physical tools detected at the mobile devices and the tasks identified at the mobile devices to generate a tool inventory and a tool compliance, the tool inventory identifying physical tools absent from the dedicated tool board and detected at the corresponding mobile device, the tool compliance identifying whether the physical tool detected at the corresponding mobile device is valid for the task selected in the augmented reality application of the corresponding mobile device;

generating an augmented reality content dataset for each mobile device, each augmented reality content dataset comprising a virtual object identifying at least one of a missing physical tool, an incorrect physical tool, and a valid physical tool based on the tool compliance; and

generating a dedicated tool board augmented reality content dataset for the dedicated tool board based on the tool inventory, the dedicated tool board augmented reality content dataset comprising a plurality of virtual objects identifying users of the mobile devices with corresponding physical tools on the dedicated tool board.

2. The method of claim 1 , wherein the dataset further comprises an identification of a user for each mobile device, an identification and a location of the physical object within a field of view of each mobile device, the task identifying a physical operation to perform on the physical object, the augmented reality application configured to generate virtual objects displayed in a transparent display of the corresponding mobile device, the virtual objects including a visual illustration of how to perform the task and how operate the physical tool related to the task on the physical object, wherein the method further comprises:

identifying dataset relevant to a corresponding mobile device and updating a local recognition content dataset at the corresponding mobile device with the relevant dataset.

3. The method of claim 1 , wherein the dedicated tool board includes a plurality of tool identifiers, the plurality of tool identifiers including a plurality of outlines displayed on the dedicated tool board, each outline corresponding to a physical tool on the dedicated tool board, each tool identifier generated for computer vision identification.

4. The method of claim 1 , further comprising:

using a depth sensor of a mobile device with the dedicated tool board within a field of view of the mobile device, to determine the physical tools present at the dedicated tool board; and

identifying physical tools absent from the dedicated tool board based depth sensor data, a shape of an outline, and a location of the outline relative to the dedicated tool board.

5. The method of claim 1 , wherein the virtual object includes a three-dimensional model of a physical tool related to the task selected at the corresponding mobile device.

6. The method of claim 1 , wherein the virtual object includes a first visual indicator to validate a first physical tool detected at the corresponding mobile device for the task selected at the corresponding mobile device.

7. The method of claim 1 , wherein the virtual object includes a second visual indicator to identify the second physical tool detected at the corresponding mobile device as an incorrect physical tool for the task selected at the corresponding mobile device.

8. The method of claim 1 , wherein the dedicated tool board augmented reality content dataset comprises at least one of:

a first virtual object identifying a user of a mobile device corresponding to a physical tool absent from the dedicated tool board;

a second virtual object comprising a visual indicator linking a physical tool present on the dedicated tool board to the first virtual object;

a third virtual object identifying a user of a mobile device corresponding to a physical tool present on the dedicated tool board; and

a fourth virtual object comprising a visual indicator of a current location of a physical tool absent from the dedicated tool board.

9. The method of claim 1 , further comprising:

communicating the augmented reality content dataset to the corresponding mobile device;

causing a display of the augmented reality content dataset in a transparent display of the corresponding mobile device;

communicating the dedicated tool board augmented reality content dataset to a mobile device with the dedicated tool board within a field of view of the mobile device; and

causing the display of the dedicated tool board augmented reality content dataset in the transparent display of the mobile device with the dedicated tool board within the field of view of the mobile device.

10. A server comprising:

a processor; and

a memory storing instructions that, when executed by the processor, configure the server to:

receive sensor data from a plurality of mobile devices having optical sensors and location sensors, the sensor data identifying a mobile physical object and a location of the mobile physical object within a predefined geographic region based on the optical and location sensors;

identify mobile physical objects that are present and absent within the predefined geographic region based on a predefined catalog of the mobile physical objects at the predefined geographic region;

generate a real-time inventory of the mobile physical objects at the predefined predefined geographic region based on the mobile physical objects that are present and absent within the predefined geographic region, the real-time inventory including the identification of the mobile physical objects and the location of the mobile physical objects within the predefined predefined geographic region;

receive datasets from the plurality of mobile devices, each dataset identifying a task selected in an augmented reality application of a corresponding mobile device, an identification of a physical tool detected at the corresponding mobile device, each mobile device including a display device;

identify physical tools present and absent at a dedicated tool board, the dedicated tool board configured to store the physical tools for the tasks of the augmented reality application;

compare an identification of the physical tools present and absent at the dedicated tool board with the physical tools detected at the mobile devices and the tasks identified at the mobile devices to generate a tool inventory and a tool compliance, the tool inventory identifying physical tools absent from the dedicated tool board and detected at the corresponding mobile device, the tool compliance identifying whether the physical tool detected at the corresponding mobile device is valid for the task selected in the augmented reality application of the corresponding mobile device;

generate an augmented reality content dataset for each mobile device, each augmented reality content dataset comprising a virtual object identifying at least one of a missing physical tool, an incorrect physical tool, and a valid physical tool based on the tool compliance; and

generate a dedicated tool board augmented reality content dataset for the dedicated tool board based on the tool inventory, the dedicated tool board augmented reality content dataset comprising a plurality of virtual objects identifying users of the mobile devices with corresponding physical tools on the dedicated tool board.

11. The server of claim 10 , wherein the dataset further comprises an identification of a user for each mobile device, an identification and a location of the physical object within a field of view of each mobile device,

the task identify a physical operation to perform on the physical object,

the augmented reality application configured to generate virtual objects displayed in a transparent display of the corresponding mobile device, the virtual objects include a visual illustration of how to perform the task and how operate the physical tool related to the task on the physical object.

12. The server of claim 10 , wherein the dedicated tool board includes a plurality of tool identifiers, the plurality of tool identifiers include a plurality of outlines displayed on the dedicated tool board, each outline corresponding to a physical tool on the dedicated tool board, each tool identifier generated for computer vision identification.

13. The server of claim 10 , wherein the instructions further configure the server to:

using a depth sensor of a mobile device with the dedicated tool board within a field of view of the mobile device, to determine the physical tools present at the dedicated tool board; and

identify physical tools absent from the dedicated tool board based depth sensor data, a shape of an outline, and a location of the outline relative to the dedicated tool board.

14. The server of claim 10 , wherein the virtual object includes a three-dimensional model of a physical tool related to the task selected at the corresponding mobile device.

15. The server of claim 10 , wherein the virtual object includes a first visual indicator to validate a first physical tool detected at the corresponding mobile device for the task selected at the corresponding mobile device, and a second a visual indicator to identify a second physical tool detected at the corresponding mobile device as an incorrect physical tool for the task selected at the corresponding mobile device.

16. The server of claim 10 , wherein the dedicated tool board augmented reality content dataset comprises at least one of:

a first virtual object identify a user of a mobile device corresponding to a physical tool absent from the dedicated tool board;

a second virtual object comprising a visual indicator link a physical tool present on the dedicated tool board to the first virtual object;

a third virtual object identify a user of a mobile device corresponding to a physical tool present on the dedicated tool board; and

a fourth virtual object comprising a visual indicator of a current location of a physical tool absent from the dedicated tool board.

17. The server of claim 10 , wherein the instructions further configure the server to:

communicate the augmented reality content dataset to the corresponding mobile device;

cause a display of the augmented reality content dataset in a transparent display of the corresponding mobile device;

communicate the dedicated tool board augmented reality content dataset to a mobile device with the dedicated tool board within a field of view of the mobile device; and

cause the display of the dedicated tool board augmented reality content dataset in the transparent display of the mobile device with the dedicated tool board within the field of view of the mobile device.

18. A non-transitory computer-readable storage medium, the computer-readable storage medium including instructions that when executed by a computer, cause the computer to:

receive sensor data from a plurality of mobile devices having optical sensors and location sensors, the sensor data identifying a mobile physical object and a location of the mobile physical object within a predefined geographic region based on the optical and location sensors;

identify mobile physical objects that are present and absent within the predefined geographic region based on a predefined catalog of the mobile physical objects at the predefined geographic region;

generate a real-time inventory of the mobile physical objects at the predefined geographic region based on the mobile physical objects that are present and absent within the predefined geographic region, the real-time inventory including the identification of the mobile physical objects and the location of the mobile physical objects within the predefined geographic region;

receive datasets from the plurality of mobile devices, each dataset identifying a task selected in an augmented reality application of a corresponding mobile device, an identification of a physical tool detected at the corresponding mobile device, each mobile device including a display device;

identify physical tools present and absent at a dedicated tool board, the dedicated tool board configured to store the physical tools for the tasks of the augmented reality application;

compare an identification of the physical tools present and absent at the dedicated tool board with the physical tools detected at the mobile devices and the tasks identified at the mobile devices to generate a tool inventory and a tool compliance, the tool inventory identifying physical tools absent from the dedicated tool board and detected at the corresponding mobile device, the tool compliance identifying whether the physical tool detected at the corresponding mobile device is valid for the task selected in the augmented reality application of the corresponding mobile device;

generate an augmented reality content dataset for each mobile device, each augmented reality content dataset comprising a virtual object identifying at least one of a missing physical tool, an incorrect physical tool, and a valid physical tool based on the tool compliance; and

generate a dedicated tool board augmented reality content dataset for the dedicated tool board based on the tool inventory, the dedicated tool board augmented reality content dataset comprising a plurality of virtual objects identifying users of the mobile devices with corresponding physical tools on the dedicated tool board.

Assignments (12)

Continuity (1)