IP Library › Granted Patent US 10,706,626
Granted Patent B1
US 10,706,626 · App. 15/374,920 · Granted Jul 7, 2020

Augmented reality procedural system

Inventors: Roger Brent (Seattle, WA); James Ashley (Lilburn, GA)
Assignee: Roger Brent
G06T19/006G06F3/167G06K9/4671G06F3/0482G06F16/22G06F40/103G06K2209/01G06T2219/024
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,706,626
App. No.
15/374,920
Filed
Dec 9, 2016
Granted
Jul 7, 2020
Kind
B1
Examiner
WU, MING HAN
Art Unit
2618
USPC
345/633
Abstract

An augmented reality device is provided to assist users in performing new or unfamiliar experimental techniques, identify materials and products utilized in a documented action set and within a work environment, identify equipment and instruments needed in the documented action set and within the work environment, assist in performing single person (autonomous) work, collaborate with other workers, and record data and observations in an electronic laboratory notebook.

Claims (40)

1. A method of guiding and documenting procedures of an action set through an augmented reality device, the method comprising:

providing the augmented reality device, operatively disposed to a user space, comprising a processor, memory, a plurality of external transducers, a plurality of internal transducers, and a visual display;

generating a graphical overlay to an ocular field of view on the visual display;

transforming an environmental signal, captured through one or both of the plurality of external transducers and the plurality of internal transducers, into an image map comprising detected objects, wherein the processor executes image processing logic, stored in the memory, to detect environmental objects within an interaction environment from the environmental signal;

recognizing materials and products from the detected objects through spatiotemporal movement analysis, comparative physical modeling, and referencing a materials database, wherein the processor is controlled by object recognition logic, stored in the memory, to recognize the materials and products from the detected objects in a series of sequential image maps;

synchronizing the graphical overlay to coincidentally align the detected objects of the image map with the environmental objects of the interaction environment to the ocular field of view through operations of the processor controlled by overlay synchronization logic;

recognizing procedural actions through contextual visual behavior recognition by tracking spatiotemporal movement of the materials and products relative to the materials and products in the series of sequential image maps and correlating to documented procedural actions in an action set database as controlled by behavior identification logic;

capturing quantifiable data in quantifying features of the materials and products through contextual feature analysis by correlating the procedural actions and an ocular line of sight to the quantifying features of the materials and products in the series of sequential image maps;

visualizing a guided interaction overlay as the graphical overlay in response to a selection input of a documented action set from the action set database;

identifying upcoming actions from the documented action set and corresponding equipment and instrument identifiers for the upcoming actions;

identifying equipment and instruments in the graphical overlay representing the corresponding equipment and instrument identifiers;

warming the corresponding equipment and instrument identifiers in the graphical overlay in proportion to their distance from a current position in the documented action set; and

providing on the augmented reality device a comparison of the action set as carried out by a user and an ideal version of the action set, in response to the user stopping motion and requesting the ideal version of the action set via the augmented reality device.

2. The method of claim 1 further comprising:

visualizing an action set outline detailing the documented procedural actions of the documented action set through the guided interaction overlay;

visualizing a list of materials, and a list of the equipment and the instruments, utilized in the documented action set through the guided interaction overlay;

displaying a graphical identifier coincidentally aligned to the detected objects in the image map corresponding to the materials, the equipment, and the instruments in the interaction environment through the guided interaction overlay; and

displaying advisement notifications through the guided interaction overlay in response to detecting an advisement and precaution alert associated with a particular documented procedural action.

3. The method of claim 2 , wherein the list of materials and the list of equipment and instruments are visualized through the guided interaction overlay for each procedure of the action set.

4. The method of claim 1 , wherein the graphical overlay is positioned between the ocular field of view and a graphical visualization of the interaction environment.

5. The method of claim 2 , further comprising:

receiving a user interaction via the augmented reality device to modify a stored representation of the action set outline.

6. The method of claim 1 , wherein the comparison comprises a comparison of the actual behavior of powders and liquids in the user space and an ideal behavior of the powders and liquids as defined in the action set.

7. The method of claim 1 , further comprising:

exchanging a field of view of the user space from a first user to a second user via the augmented reality device.

8. The method of claim 1 , further comprising:

a user pouring liquid from a container to generate droplets;

providing feedback to the user via the augmented reality device to modify a speed of generating the droplets.

9. The method of claim 8 , the feedback comprising:

displaying on the augmented reality device an angle of orientation for the container.

10. The method of claim 1 , further comprising:

calibrating user force on a container via a force feedback sensor; and

applying the calibration to determine a user-initiated disturbance to material in the container.

11. The method of claim 1 , further comprising:

determining characteristics of powdered or granular materials based on an angle of repose of a container of the powdered or granular materials as detected via the augmented reality device.

12. The method of claim 11 , further comprising:

receiving hand gestures of a user via the augmented reality device; and

applying the hand gestures to identify boundaries of the powdered or granular materials.

13. The method of claim 1 , further comprising:

warming the corresponding equipment and instrument identifiers in the graphical overlay in proportion to their temporal relevance from a current position in the documented action set.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 4, 2021
From: IAS MACHINE LLC
To: LABLIGHTAR INC.
Reel/Frame 056439/0749 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 28, 2020
From: ASHLEY, JAMES; BRENT, ROGER
To: IAS MACHINE, LLC
Reel/Frame 053898/0768 →
Continuity (1)
Provisional Application 62265255 · Dec 9, 2015
Cited By (6)
US 12,368,503 US 12,567,214 US 12,587,274 US 12,603,701 US 12,627,372 US 12,638,400