IP Library Granted Patent US 11,656,680
Granted Patent B2
US 11,656,680 · App. 17/012,309 · Granted May 23, 2023

Technique for controlling virtual image generation system using emotional states of user

Inventors: George Alistair Sanger (Coronado, CA); Samuel A. Miller (Hollywood, FL); Graeme John Devine (Rockwall, TX)
Assignee: Magic Leap, Inc.
G06F3/012A63F13/21A63F13/212A63F13/52A63F13/65A63F13/822G02B27/017G06F3/011G06F3/0304G06F16/56G06F16/5866G06T19/006G06V20/20G06V40/175G02B27/0172G02B2027/014G02B2027/0134G02B2027/0138G02B2027/0187G06F3/013G06F3/015G06F2203/011
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,656,680
App. No.
17/012,309
Granted
May 23, 2023
Kind
B2
Abstract

A method of operating a virtual image generation system comprises allowing an end user to interact with a three-dimensional environment comprising at least one virtual object, presenting a stimulus to the end user in the context of the three-dimensional environment, sensing at least one biometric parameter of the end user in response to the presentation of the stimulus to the end user, generating biometric data for each of the sensed biometric parameter(s), determining if the end user is in at least one specific emotional state based on the biometric data for the each of the sensed biometric parameter(s), and performing an action discernible to the end user to facilitate a current objective at least partially based on if it is determined that the end user is in the specific emotional state(s).

Claims (35)

1. A method of operating an augmented reality (AR) display system that includes a wearable display device, the method comprising:

responsive to presenting a stimulus to a user through the wearable display device, sensing at least one biometric parameter of the user, the at least one biometric parameter sensed using at least one sensor that is attached to the wearable display device;

generating biometric data for each of the at least one sensed biometric parameter;

determining at least one emotional state of the user based on the biometric data for the each of the at least one sensed biometric parameter; and

presenting at least one additional virtual object through the AR display system based at least partly on determining that the user is in the at least one emotional state, wherein the at least one additional virtual object is selected to evoke another emotional state in the user different from the at least one emotional state of the user,

wherein determining at least one emotional state of the user comprises

retrieving a reference biometric data from a custom emotional state profile of the user, the custom emotional state profile of the user comprising biometric data indicating respective emotional states experienced by the user, and

comparing the generated biometric data to the reference biometric data correlated to the at least one emotional state.

2. The method of claim 1 , further comprising determining the at least one emotional state of the user for a duration, wherein the at least one additional virtual object is presented through the AR display system based at least partially on determining that the user is in the at least one emotional state for the duration.

3. The method of claim 1 , wherein the at least one emotional state comprises a plurality of different emotional states.

4. The method of claim 1 , wherein the at least one biometric parameter of the user is sensed at a plurality of different times in response to the presenting the stimulus, and the biometric data is generated at these different times.

5. The method of claim 1 , wherein the stimulus is visually presented to the user through the wearable display device.

6. The method of claim 1 , wherein the at least one sensed biometric parameter comprises a plurality of different sensed biometric parameters, and

wherein determining the at least one emotional state of the user comprises performing a pattern recognition analysis on the generated biometric data.

7. The method of claim 1 , wherein the at least one sensed biometric parameter comprises at least one facial expression.

8. The method of claim 7 , wherein the least one facial expression is one or both of an attitude of a mouth and crow's feet around eyes of the user, and

wherein the at least one emotional state comprises happiness.

9. The method of claim 1 , wherein the at least one sensed biometric parameter comprises at least one of a facial expression, hunching of shoulders, respiration rate, heart rate, body temperature, blood pressure, frequency and/or location of hand movements, frequency and/or location of body twitches, and elapsed time between eye movements.

10. The method of claim 1 , wherein the at least one sensed biometric parameter comprises at least one micro-expression.

11. The method of claim 1 , wherein the at least one emotional state comprises at least one of anger, contempt, disgust, fear, happiness, sadness, surprise, confusion, shame, attentiveness, exhaustion, relaxation, frustration, boredom, embarrassment.

12. The method of claim 1 , wherein the generated biometric data for one of the at least one sensed biometric parameter is a biometric scalar data value, the reference biometric data comprises a reference biometric value range, and comparing the generated biometric data to the reference biometric data comprises determining whether the biometric scalar data value falls within the reference biometric value range.

13. The method of claim 1 , wherein the generated biometric data for one of the at least one sensed biometric parameter is a biometric multi-dimensional data vector, the reference biometric data comprises a reference biometric multi-dimensional data vector, and comparing the generated biometric data to the reference biometric data comprises performing a correlation function between the generated biometric multi-dimensional data vector and the reference biometric multi-dimensional data vector.

14. The method of claim 1 , further comprising:

rendering a plurality of synthetic image frames of a three-dimensional environment; and

sequentially displaying the plurality of synthetic image frames to the user through wearable display device,

wherein the synthetic image frames are projected from a transparent display surface in a field of view of the user via a frame structure mounted to a head of the user, and

wherein the synthetic image frames are superimposed over a real scene visualized by the user.

15. The method of claim 1 , wherein the AR display system also includes a speaker coupled to the wearable display device, and

wherein the stimulus is aurally presented through the speaker.

16. The method of claim 1 , wherein the AR display system also includes a mechanical actuator, and

wherein the stimulus is tactilely presented through the mechanical actuator.

17. The method of claim 1 , wherein the at least one sensor comprises a rearward facing camera coupled to the wearable display device, and

wherein the at least one biometric parameter comprises a facial movement of the user.

18. The method of claim 1 , wherein the at least one sensor comprises an accelerometer coupled to a body part of the user, and

wherein the at least one biometric parameter comprises a movement of the body part of the user.

Assignments (3)
SECURITY INTEREST Recorded Oct 15, 2025
From: MAGIC LEAP, INC.; MENTOR ACQUISITION ONE, LLC; MOLECULAR IMPRINTS, INC.
To: CITIBANK, N.A., AS COLLATERAL AGENT
Reel/Frame 073109/0238 →
SECURITY INTEREST Recorded May 24, 2022
From: MOLECULAR IMPRINTS, INC.; MENTOR ACQUISITION ONE, LLC; MAGIC LEAP, INC.
To: CITIBANK, N.A., AS COLLATERAL AGENT
Reel/Frame 060338/0665 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 4, 2020
From: SANGER, GEORGE ALISTAIR; MILLER, SAMUEL A.; DEVINE, GRAEME JOHN
To: MAGIC LEAP, INC.
Reel/Frame 053694/0387 →
Continuity (4)
Continuation 16713434 · Dec 13, 2019
Continuation 15655563 · Jul 20, 2017
Provisional Application 62364957 · Jul 21, 2016
Related Publication 20200401216A1 · Dec 24, 2020