IP Library › Granted Patent US 11,756,272
Granted Patent B2
US 11,756,272 · App. 17/823,015 · Granted Sep 12, 2023

Somatic and somatosensory guidance in virtual and augmented reality environments

Inventors: Roger Brent (Seattle, WA); John Max Kellum (Seattle, WA); James Ashley (Lilburn, GA)
Assignee: LabLightAR, Inc.
G06T19/006G06F3/011G06F9/453G06T19/20G09B19/003G06T2219/2012G06T2219/2016
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,756,272
App. No.
17/823,015
Filed
Aug 29, 2022
Granted
Sep 12, 2023
Kind
B2
Examiner
GRAY, RYAN M
Art Unit
2611
USPC
345/633
Abstract

A method of operating a procedural training user interface system involves displaying an interactive guided process of a first user using at least one augmented reality (AR) layer through an AR device worn by a second user, where a representation of first user hands is displayed. The second user interactions may be detected during the interactive guided process, where the second user attempts to superimpose second user hands on the representation of the first user hands in the at least one AR layer. The interactive guided process of the second user may then be displayed using the AR layer through an AR device on the first user and the AR device on the second user. If the first user hands and the second user hands are not superimposed in the AR layer, the first user or the second user may be notified to take corrective action.

Claims (54)

1. A method of operating a procedural guidance system, the method comprising:

detecting first user interactions of a first user during an interactive guided process;

displaying to a second user, using at least one augmented reality (AR) layer, one or more representations of one or more first user hands of the first user carrying out the interactive guided process;

detecting second user interactions of the second user during the interactive guided process, wherein the second user attempts to superimpose one or more representations of one or more second user hands with the one or more representations of the one or more first user hands in the at least one AR layer;

determining if the one or more representations of the one or more first user hands and the one or more representations of the one or more second user hands are superimposed in the at least one AR layer;

in response to determining none of the representations of the one or more first user hands and the one or more second user hands being superimposed in the at least one AR layer, generating corrective action signals for the first user, the second user, or both; and

In response to determining at least one or more fingertips in a representation of a second user hand that are in contact with an object being superimposed on one or more corresponding fingertips in an additional representation of a corresponding first user hand in the at least one AR layer without other portions of the first user hand and the second user in each representation being superimposed, generating a continue signal for at least the second user or permitting a procedure in the interactive guided process to advance to a next step.

2. The method of claim 1 , wherein the second user hand of the second user is larger or smaller than the first user hand of the first user.

3. The method of claim 1 , further comprising:

detecting the first user interactions from a recording of the interactive guided process.

4. The method of claim 1 , further comprising:

detecting the first user interactions from a live transmission of the interactive guided process.

5. The method of claim 1 , further comprising:

scaling the representations of the first user hands to fit the representations of the second user hands, or vice versa.

6. The method of claim 1 , further comprising:

determining a size relationship between the first user hands and the second user hands; and

scaling a projection volume for the first user hands in the AR layer according to the size relationship.

7. The method of claim 6 , further comprising:

superimposing fingertips of the second user hands on fingertips of the first user hands while other portions of the first user hands and the second user hands are not superimposed.

8. The method of claim 1 , wherein at least one of the representations of the first user hands or the second user hands comprises an internal anatomical map of a hand.

9. The method of claim 8 , wherein anatomical features of the hand in a corresponding representation are color coded.

10. The method of claim 9 , wherein one or more colors in the corresponding representation depict one or both of muscle tension and imminent movement of the anatomical features.

11. A procedural guidance system comprising:

an augmented reality device; and

logic configured to:

detect first user interactions of a first user during an interactive guided process;

display to a second user utilizing the augmented reality device one or more representations of one or more first user hands of the first user carrying out the interactive guided process;

detect second user interactions of the second user during the interactive guided process, wherein the second user attempts to superimpose one or more representations of one or more second user hands with the one or more representations of the one or more first user hands in a display generated by the augmented reality device;

determine if the one or more representations of the one or more first user hands and the one or more representations of the one or more second user hands are superimposed in the display; and

in response to determining none of the representations of the one or more first user hands and the one or more second user hands being superimposed in the display, generate corrective action signals for the first user, the second user, or both; and

In response to determining at least one or more fingertips in a representation of a second user hand that are in contact with an object being superimposed on one or more corresponding fingertips in an additional representation of a corresponding first user hand in the display without other portions of the first user hand and the second user in each representation being superimposed, generate a continue signal for at least the second user or permitting a procedure in the interactive guided process to advance to a next step.

12. The system of claim 11 , the logic further configured to:

detect the first user interactions from a recording of the interactive guided process.

13. The system of claim 11 , the logic further configured to:

detect the first user interactions from a live transmission of the interactive guided process.

14. The system of claim 11 , the logic further configured to:

scale the representations of the first user hands to fit the representations of the second user hands, or vice versa.

15. The system of claim 11 , the logic further configured to:

determine a size relationship between the first user hands and the second user hands; and

scale a projection volume for the first user hands in the display according to the size relationship.

16. The system of claim 15 , the logic further configured to:

superimpose fingertips of the second user hands on fingertips of the first user hands while other portions of the first user hands and the second user hands are not superimposed.

17. The system of claim 11 , wherein at least one of the representations of the first user hands or the second user hands comprise an internal anatomical map of a hand.

18. The system of claim 17 , wherein anatomical features of the hand in a corresponding the representation are color coded.

19. The system of claim 18 , wherein one or more colors in the corresponding representation depict one or both of muscle tension and imminent movement of the anatomical features.

20. A procedural guidance system comprising:

an augmented reality device; and

logic configured to:

detect first user interactions of a first user during an interactive guided process;

display to a second user utilizing the augmented reality device one or more representations of one or more first user hands of the first user carrying out the interactive guided process;

detect second user interactions of the second user during the interactive guided process, wherein the second user attempts to superimpose one or more representations of one or more second user hands with the one or more representations of the one or more first user hands in a display generated by the augmented reality device;

determine if the one or more representations of the one or more first user hands and the one or more representations of the one or more second user hands are superimposed in the display; and

in response to determining none of the representations of the one or more first user hands and the one or more second user hands being superimposed in the display, generate corrective action signals for the first user, the second user, or both,

wherein at least one of the representations of the first user hands or the second user hands comprise an internal anatomical map of a hand.

Assignments (3)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jul 21, 2023
From: BRENT, ROGER; ASHLEY, JAMES
To: IAS MACHINE, LLC
Reel/Frame 064340/0702 →
ADVISOR AGREEMENT Recorded Jul 21, 2023
From: KELLUM, JOHN M.
To: IAS MACHINE, LLC
Reel/Frame 064355/0777 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 8, 2023
From: IAS MACHINE, LLC
To: LABLIGHTAR INC.
Reel/Frame 062925/0959 →
Continuity (2)
Provisional Application 63237992 · Aug 27, 2021
Related Publication 20230068660A1 · Mar 2, 2023
Cited By (4)
US 12,306,878 US 12,354,500 US 12,493,654 US 12,718,505