IP Library › Granted Patent US 12,236,920
Granted Patent B2
US 12,236,920 · App. 16/939,635 · Granted Feb 25, 2025

Augmented reality sessions responsive to eye behavior

Inventors: Robert Huntington Grant (Marietta, GA); Juan C. Lopez (Tampa, FL); Alexander DeFarlo (Armonk, NY); Zachary A. Silverstein (Jacksonville, FL)
Assignee: International Business Machines Corporation
G09G5/36A61B5/163A61B5/165G06K9/00604A61B3/112A61B3/113A61B2090/365A61B2505/09G09G2354/00
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,236,920
App. No.
16/939,635
Granted
Feb 25, 2025
Kind
B2
Abstract

A method, system, and computer program product for generating augmented reality sessions based on eye behavior of a user is provided. The method detects a set of eye characteristics of a user. In response to detecting the set of eye characteristics, presentation of an augmented reality session is initiated using an augmented reality device. The method detects a change to at least one eye characteristic of the set of eye characteristics of the user. In response to detecting the change, the method determines the change to the at least one eye characteristic indicates a reduced cognitive state of the user. The augmented reality state being presented to the user is modified in response to determining the change indicates the reduced cognitive state.

Claims (69)

1. A computer-implemented method, comprising:

detecting, using an image capture device of an augmented reality device, a set of eye characteristics of a user;

generating a session log for a set of historic augmented reality sessions presented to the user, the session log including a set of session characteristics and eye characteristics of the user occurring during each historic augmented reality session;

based on the eye characteristics of the user occurring during the set of historic augmented reality sessions, generating a set of characteristic thresholds for the user;

in response to detecting the set of eye characteristics, initiating presentation of an augmented reality session using the augmented reality device;

detecting a change to at least one eye characteristic of the set of eye characteristics of the user;

in response to detecting the change and based on the set of characteristic thresholds for the user, determining the change to the at least one eye characteristic indicates a reduced cognitive state of the user; and

in response to determining the change indicates the reduced cognitive state, modifying the augmented reality session being presented to the user.

2. The computer-implemented method of claim 1 , wherein the image capture device of the augmented reality device is a first image capture device, the method further comprising:

detecting, using a second image capture device of the augmented reality device, a session administrator;

receiving session characteristics from a computing device associated with the session administrator; and

wherein the augmented reality session is initiated in response to detecting the set of eye characteristics and receiving session characteristics from the computing device.

3. The computer-implemented method of claim 1 , wherein determining the change indicates the reduced cognitive state further comprises:

determining a standard cognitive state for the user, the standard cognitive state associated with a set of characteristic thresholds of the user;

in response to detecting the change to the at least one eye characteristic, accessing the standard cognitive state for the user; and

determining the change to the at least one eye characteristic exceeds a characteristic threshold of the set of characteristic thresholds.

4. The computer-implemented method of claim 1 , wherein the augmented reality session is a current augmented reality session, and wherein modifying the augmented reality session further comprises:

identifying a current session topic from session characteristics of the current augmented reality session;

based on identifying the session topic, identifying, from the session log, a subset of historic augmented reality sessions with historic session topics matching the session topic;

selecting a positive historic augmented reality session from the subset of historic augmented reality sessions, the positive historic augmented reality session being associated with a cognitive state greater than the reduced cognitive state; and

initiating presentation of at least a portion of the positive historic augmented reality session within the current augmented reality session.

5. The computer-implemented method of claim 1 , further comprising:

in response to completing the augmented reality session, storing session characteristics of the augmented reality session and eye characteristics of the user in the session log.

6. A system, comprising:

one or more processors; and

a computer-readable storage medium, coupled to the one or more processors, storing program instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:

detecting, using an image capture device of an augmented reality device, a set of eye characteristics of a user;

generating a session log for a set of historic augmented reality sessions presented to the user, the session log including a set of session characteristics and eye characteristics of the user occurring during each historic augmented reality session;

based on the eye characteristics of the user occurring during the set of historic augmented reality sessions, generating a set of characteristic thresholds for the user;

in response to detecting the set of eye characteristics, initiating presentation of an augmented reality session using the augmented reality device;

detecting a change to at least one eye characteristic of the set of eye characteristics of the user;

in response to detecting the change and based on the set of characteristic thresholds for the user, determining the change to the at least one eye characteristic indicates a reduced cognitive state of the user; and

in response to determining the change indicates the reduced cognitive state, modifying the augmented reality session being presented to the user.

7. The system of claim 6 , wherein the image capture device of the augmented reality device is a first image capture device, the operations further comprise:

detecting, using a second image capture device of the augmented reality device, a session administrator;

receiving session characteristics from a computing device associated with the session administrator; and

wherein the augmented reality session is initiated in response to detecting the set of eye characteristics and receiving session characteristics from the computing device.

8. The system of claim 6 , wherein determining the change indicates the reduced cognitive state further comprises:

determining a standard cognitive state for the user, the standard cognitive state associated with a set of characteristic thresholds of the user;

in response to detecting the change to the at least one eye characteristic, accessing the standard cognitive state for the user; and

determining the change to the at least one eye characteristic exceeds a characteristic threshold of the set of characteristic thresholds.

9. The system of claim 6 , wherein the augmented reality session is a current augmented reality session, and wherein modifying the augmented reality session further comprises:

identifying a current session topic from session characteristics of the current augmented reality session;

based on identifying the session topic, identifying, from the session log, a subset of historic augmented reality sessions with historic session topics matching the session topic;

selecting a positive historic augmented reality session from the subset of historic augmented reality sessions, the positive historic augmented reality session being associated with a cognitive state greater than the reduced cognitive state; and

initiating presentation of at least a portion of the positive historic augmented reality session within the current augmented reality session.

10. The system of claim 6 , wherein the operations further comprise:

in response to completing the augmented reality session, storing session characteristics of the augmented reality session and eye characteristics of the user in the session log.

11. A computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions being executable by one or more processors to cause the one or more processors to perform operations comprising:

detecting, using an image capture device of an augmented reality device, a set of eye characteristics of a user;

generating a session log for a set of historic augmented reality sessions presented to the user, the session log including a set of session characteristics and eye characteristics of the user occurring during each historic augmented reality session;

based on the eye characteristics of the user occurring during the set of historic augmented reality sessions, generating a set of characteristic thresholds for the user;

in response to detecting the set of eye characteristics, initiating presentation of an augmented reality session using the augmented reality device;

detecting a change to at least one eye characteristic of the set of eye characteristics of the user;

in response to detecting the change and based on the set of characteristic thresholds for the user, determining the change to the at least one eye characteristic indicates a reduced cognitive state of the user; and

in response to determining the change indicates the reduced cognitive state, modifying the augmented reality session being presented to the user.

12. The computer program product of claim 11 , wherein the image capture device of the augmented reality device is a first image capture device, the operations further comprise:

detecting, using a second image capture device of the augmented reality device, a session administrator;

receiving session characteristics from a computing device associated with the session administrator; and

wherein the augmented reality session is initiated in response to detecting the set of eye characteristics and receiving session characteristics from the computing device.

13. The computer program product of claim 11 , wherein determining the change indicates the reduced cognitive state further comprises:

determining a standard cognitive state for the user, the standard cognitive state associated with a set of characteristic thresholds of the user;

in response to detecting the change to the at least one eye characteristic, accessing the standard cognitive state for the user; and

determining the change to the at least one eye characteristic exceeds a characteristic threshold of the set of characteristic thresholds.

14. The computer program product of claim 11 , wherein the augmented reality session is a current augmented reality session, and wherein modifying the augmented reality session further comprises:

identifying a current session topic from session characteristics of the current augmented reality session;

based on identifying the session topic, identifying, from the session log, a subset of historic augmented reality sessions with historic session topics matching the session topic;

selecting a positive historic augmented reality session from the subset of historic augmented reality sessions, the positive historic augmented reality session being associated with a cognitive state greater than the reduced cognitive state; and

initiating presentation of at least a portion of the positive historic augmented reality session within the current augmented reality session.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jul 27, 2020
From: GRANT, ROBERT HUNTINGTON; LOPEZ, JUAN C.; DEFARLO, ALEXANDER; SILVERSTEIN, ZACHARY A.
To: INTERNATIONAL BUSINESS MACHINES CORPORATION
Reel/Frame 053318/0805 →
Continuity (1)
Related Publication 20220028356A1 · Jan 27, 2022
References Cited (28)
US 6231187B1 · Munoz et al. · 2001 [cited by applicant]
US 9823737B2 · Mazed et al. · 2017 [cited by applicant]
US 10332315B2 · Samec et al. · 2019 [cited by applicant]
US 20040181168A1 · Plant et al. · 2004 [cited by applicant]
US 20130278899A1 · Waldorf et al. · 2013 [cited by applicant]
US 20130321772A1 · White · 2013 [cited by examiner]
US 20140255888A1 · Stack · 2014 [cited by examiner]
US 20150097772A1 · Starner · 2015 [cited by examiner]
US 20160007921A1 · Galea et al. · 2016 [cited by applicant]
US 20160085302A1 · Publicover · 2016 [cited by examiner]
US 20160229412A1 · Gordon · 2016 [cited by examiner]
US 20160262608A1 · Krueger · 2016 [cited by applicant]
US 20170135577A1 · Komogortsev · 2017 [cited by examiner]
US 20170365101A1 · Samec · 2017 [cited by examiner]
US 20190009049A1 · Candy · 2019 [cited by applicant]
US 20190246969A1 · Thomas · 2019 [cited by examiner]
US 20190287309A1 · Samec et al. · 2019 [cited by applicant]
US 20190307350A1 · Sridhar et al. · 2019 [cited by applicant]
US 20190307384A1 · Baeuerle · 2019 [cited by examiner]
US 20200029802A1 · Lane · 2020 [cited by examiner]
US 20200187860A1 · Myslinski · 2020 [cited by examiner]
WO 2017186721A1 · 2017 [cited by applicant]
Orlosky et al., “Emulation of Physician Tasks in Eye-Tracked Virtual Reality for Remote Diagnosis of Neurodegenerative Disease,” in IEEE Transactions on Visualization and Computer Graphics, vol. 23, No. 4, Apr. 2017, 10… [cited by applicant]
Molitor et al., “Eye Movements in Alzheimer's Disease”, J Alzheimers Dis. 2015 ; 44(1), doi:10.3233/JAD-141173, 18 pages, available in PMC Mar. 1, 2017. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5332166/. [cited by applicant]
IBM Research, IoT and Wearables, printed Feb. 21, 2020, 6 pages. https://www.research.ibm.com/haifa/dept/services/wearable.shtml. [cited by applicant]
“Activities at Home”, Alzheimer's Association, © 2019 Alzheimer's Association, Sep. 2019, 12 pages. [cited by applicant]
Schroder et al., “Deep Learning for Action Recognition in Augmented Reality Assistance Systems”, SIGGRAPH '17 Posters, Los Angeles, CA, DOI: 10.1145/3102163.3102191, 2 pages. [cited by applicant]
Mell et al., “The NIST Definition of Cloud Computing”, Recommendations of the National Institute of Standards and Technology, Special Publication 800-145, Sep. 2011, 7 pages. [cited by applicant]