IP Library Granted Patent US 11,467,662
Granted Patent B1
US 11,467,662 · App. 17/370,745 · Granted Oct 11, 2022

Identifying object of user focus with eye tracking and visually evoked potentials

Inventors: Yu-Te Wang (San Diego, CA); Mark Allan Chevillet (Redwood City, CA)
Assignee: Meta Platforms, Inc.
G06F3/013A61B5/291A61B5/378A61B5/6803G06T3/60G06T7/70G06T11/001G06F3/015G06T2207/30201
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,467,662
App. No.
17/370,745
Granted
Oct 11, 2022
Kind
B1
Abstract

A brain computer interface system includes a wearable interface, an eye tracking device, and a client device for determining what object a user is looking at on an electronic display. The client device determines a region on the electronic display based on an estimated user gaze direction received from the eye tracking device. For each virtual object in the gaze region, the client device displays a visual stimulus with a unique frequency. The client device receives from the wearable interface an electrical potential signal measured at the user's brain and evoked by a visual stimulus on the electronic display. The client device identifies the object in the gaze region with a stimulus frequency matching a frequency derived from the potential signal, and executes instructions relating to the object.

Claims (53)

1. A system comprising:

an interface for a head region of a user, the interface including a plurality of electrodes operable to measure signals evoked by visual stimuli that the user is looking at, wherein the wearable interface is operable to transmit data derived from the signals;

an electronic display including a plurality of pixels operable to display a plurality of objects using the plurality of pixels; and

a client device operable to receive data from the interface, the client device storing instructions that when executed by the client device cause the client device to perform steps including:

determine a region on the electronic display approximating where a user is looking, the electronic display including a plurality of pixels representing a plurality of objects, wherein the region includes a portion of the plurality of pixels;

identify a subset of objects of the plurality of objects represented by pixels of the portion of the plurality of pixels within the region on the electronic display;

responsive to identifying the subset of objects, display a visual stimulus for each object of the subset of objects on the electronic display by altering the pixels representing each object, wherein each visual stimulus has a unique frequency relative to the visual stimuli for other objects of the subset of objects;

receive a visually evoked signal from the wearable interface;

compare a frequency derived from the visually evoked signal to the frequency of each visual stimulus of the subset of objects;

identify, based on the comparison, an object of the subset of objects corresponding to a visual stimulus having a frequency that matches the frequency derived from the visually evoked signal; and

perform an action associated with the identified object.

2. The system of claim 1 , wherein the electronic display and the interface are components of a virtual reality headset.

3. The system of claim 1 , wherein the electronic display and the interface are components of an augmented reality headset.

4. The system of claim 1 , wherein the instructions when executed by the client device further cause the client device to perform steps including:

receive a user gaze direction captured by an eye tracking device, wherein the gaze direction approximates where the user is looking; and

determine the region on the electronic display using the user gaze direction.

5. The system of claim 1 , wherein one or more electrodes of the plurality of electrodes are dry electrodes.

6. The system of claim 1 , wherein the plurality of electrodes comprises a reference electrode and a recording electrode, and wherein the interface measures the visually evoked signal between the recording electrode and the reference electrode.

7. The system of claim 1 , wherein the visually evoked signal is a plurality of steady state visually evoked potentials measured by a plurality of electrodes of the interface.

8. The system of claim 1 , wherein the instructions when executed by the client device further cause the client device to perform steps including:

determine a number of objects in the subset of objects;

select a set of frequencies based on a refresh rate of the electronic display and the determined number of objects;

assign a different frequency from the selected set of frequencies to each of the objects in the subset of objects; and

alter the pixels representing each of the objects in subset of objects based on the assigned frequencies.

9. The system of claim 1 , wherein the instructions when executed by the client device further cause the client device to perform steps including:

responsive to identifying an object of the subset of objects corresponding to the visual stimulus, stop the display of the visual stimulus.

10. A method performed by a client device comprising:

determining a region on an electronic display approximating where a user is looking, the electronic display including a plurality of pixels representing a plurality of objects, wherein the region includes a portion of the plurality of pixels;

identifying a subset of objects of the plurality of objects represented by pixels of the portion of the plurality of pixels within the region on the electronic display;

responsive to identifying the subset of objects, displaying a visual stimulus for each object of the subset of objects on the electronic display by altering the pixels representing each object, wherein each visual stimulus has a unique frequency relative to the visual stimuli for other objects of the subset of objects;

receiving a visually evoked signal measured by an interface for a head region of the user;

comparing a frequency derived from the visually evoked signal to the frequency of each visual stimulus of the subset of objects;

identifying, based on the comparison, an object of the subset of objects corresponding to a visual stimulus having a frequency that matches the frequency derived from the visually evoked signal; and

perform an action associated with the identified object.

11. The method of claim 10 , wherein determining the region on the electronic display comprises:

receiving a gaze direction for the user captured by an eye tracking device; and

determining the region based on the gaze direction.

12. The method of claim 10 , wherein the region on the electronic display is determined based on a field of view of the user.

13. The method of claim 10 , wherein the visually evoked signal is a plurality of steady state visually evoked potentials measured by a plurality of electrodes on the interface.

14. The method of claim 10 , wherein displaying the visual stimulus for each object comprises:

determining a number of objects in the subset of objects;

selecting a set of frequencies based on a refresh rate of the electronic display and the determined number of objects;

assigning a different frequency from the selected set of frequencies to each of the objects in the subset of objects; and

altering the pixels representing each of the objects in the subset of objects based on the assigned frequencies.

15. The method of claim 10 , further comprising:

responsive to identifying an object of the subset of objects corresponding to the visual stimulus, stopping the display of the visual stimulus.

16. The method of claim 10 , wherein displaying a visual stimulus comprises altering the pixels representing one or more objects of the subset of objects to display a pattern including one or more of a solid color, a polychromatic spiral, or a checkerboard.

17. The method of claim 10 , wherein the frequency of one or more of the visual stimuli corresponds to a rate at which the pixels representing one or more objects of the subset of objects corresponding to the one or more visual stimuli alternate between a first color and a second color.

18. The method of claim 10 , wherein the frequency of one or more of the visual stimuli corresponds to a rate at which the one or more visual stimuli rotate.

19. The method of claim 10 , wherein determining the derived frequency comprises:

computing, for each template of a set of signal templates, a correlation between the visually evoked signal and the template, wherein each template corresponds to a distinct frequency; and

selecting a frequency corresponding to a template from the set of signal templates having a highest correlation as the derived frequency.

20. The method of claim 10 , wherein the frequency derived from the visually evoked signal is determined based on a plurality of signals measured at a plurality of electrodes on the interface.

Assignments (1)
CHANGE OF NAME Recorded Nov 18, 2021
From: FACEBOOK, INC.
To: META PLATFORMS, INC.
Reel/Frame 058897/0824 →
Continuity (1)
Continuation 16666282 · Oct 28, 2019