IP Library Granted Patent US 11,093,033
Granted Patent B1
US 11,093,033 · App. 16/666,282 · Granted Aug 17, 2021

Identifying object of user focus with eye tracking and visually evoked potentials

Inventors: Yu-Te Wang (San Diego, CA); Mark Allan Chevillet (Redwood City, CA)
Assignee: Facebook, Inc.
G06F3/013A61B5/291A61B5/378A61B5/6803G06T3/60G06T7/70G06T11/001G06F3/015G06T2207/30201
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,093,033
App. No.
16/666,282
Granted
Aug 17, 2021
Kind
B1
Abstract

A brain computer interface system includes a wearable interface, an eye tracking device, and a client device for determining what object a user is looking at on an electronic display. The client device determines a region on the electronic display based on an estimated user gaze direction received from the eye tracking device. For each virtual object in the gaze region, the client device displays a visual stimulus with a unique frequency. The client device receives from the wearable interface an electrical potential signal measured at the user's brain and evoked by a visual stimulus on the electronic display. The client device identifies the object in the gaze region with a stimulus frequency matching a frequency derived from the potential signal, and executes instructions relating to the object.

Claims (55)

1. A system comprising:

an eye tracking device including an eye imaging sensor, wherein the eye tracking device is operable to transmit data derived from the eye imaging sensor;

a wearable interface worn at a head region of a user, the wearable interface including a plurality of electrodes operable to measure signals evoked by visual stimuli that the user is looking at, wherein the wearable interface is operable to transmit data derived from the signals;

an electronic display including a plurality of pixels operable to display a plurality of objects using the plurality of pixels; and

a client device operable to receive data from the eye tracking device and the wearable interface, the client device configured to:

determine, using the data received from the eye tracking device, a gaze region of the user on the electronic display;

identify a subset of objects of the plurality of objects represented by pixels of the plurality of pixels within the gaze region on the electronic display;

responsive to identifying the subset of objects, display a visual stimulus for each object of the subset of objects on the electronic display by altering the pixels representing each object, wherein each visual stimulus has a unique frequency relative to the visual stimuli for other objects of the subset of objects;

receive a visually evoked signal from the wearable interface;

compare a frequency derived from the visually evoked signal to the frequency of each visual stimulus of the subset of objects;

identify, based on the comparison, an object of the subset of objects corresponding to a visual stimulus having a frequency that matches the frequency derived from the visually evoked signal; and

execute a set of instructions associated with the identified object.

2. The system of claim 1 , wherein the electronic display, the eye tracking device, and the wearable interface are components of a virtual reality headset.

3. The system of claim 1 , wherein the electronic display, the eye tracking device, and the wearable interface are components of an augmented reality headset.

4. The system of claim 1 , wherein the client device is further configured to:

determine a user gaze direction using the data received from the eye tracking device, wherein the gaze direction approximates where the user is looking;

map the gaze direction to a position on the electronic display; and

determine the gaze region using the position on the electronic display, the gaze region encompassing the position on the electronic display.

5. The system of claim 1 , wherein one or more electrodes of the plurality of electrodes are dry electrodes.

6. The system of claim 1 , wherein the plurality of electrodes comprises a reference electrode and a recording electrode, the wearable interface measures the visually evoked signal between the recording electrode and the reference electrode.

7. The system of claim 1 , wherein the gaze region is determined based on a field of view of the user.

8. The system of claim 1 , wherein the visually evoked signal is a plurality of steady state visually evoked potentials measured by a plurality of electrodes of the wearable interface.

9. The system of claim 1 , wherein the client device is further configured to:

determine a number of objects in the subset of objects;

select a set of frequencies based on a refresh rate of the electronic display and the determined number of objects;

assign a different frequency from the selected set of frequencies to each of the objects in the subset of objects; and

alter the pixels representing each of the objects in subset of objects based on the assigned frequencies.

10. The system of claim 1 , wherein the client device is further configured to:

responsive to identifying an object of the subset of objects corresponding to the visual stimulus, stop the display of the visual stimulus.

11. A method performed by a client device comprising:

receiving a user gaze direction from an eye tracking device, wherein the gaze direction approximates where a user of the client device is looking;

mapping the gaze direction to a position on an electronic display including a plurality of pixels, the plurality of pixels representing a plurality of objects;

determining a gaze region on the electronic display encompassing the mapped position;

identifying a subset of objects of the plurality of objects represented by pixels of the plurality of pixels within the gaze region on the electronic display;

responsive to identifying the subset of objects, displaying a visual stimulus for each object of the subset of objects by altering the pixels representing each object, wherein each visual stimulus has a unique frequency relative to the visual stimuli for other objects of the subset of objects;

receiving a visually evoked signal from a wearable interface worn by a user, the visually evoked signal measured by the wearable interface at a head region of the user;

comparing a frequency derived from the visually evoked signal to the frequency of each visual stimulus of the subset of objects;

identifying, based on the comparison, an object of the subset of objects corresponding to a visual stimulus having a frequency of the object that matches the frequency derived from the visually evoked signal; and

executing a set of instructions on the client device associated with the identified object.

12. The method of claim 11 , wherein the gaze region is determined based on a field of view of the user.

13. The method of claim 11 , wherein the visually evoked signal is a plurality of steady state visually evoked potentials measured by a plurality of electrodes on the wearable interface.

14. The method of claim 11 , wherein displaying the visual stimulus for each object comprises:

determining a number of objects in the subset of objects;

selecting a set of frequencies based on a refresh rate of the electronic display and the determined number of objects;

assigning a different frequency from the selected set of frequencies to each of the objects in the subset of objects; and

altering the pixels representing each of the objects in the subset of objects based on the assigned frequencies.

15. The method of claim 11 , further comprising:

responsive to identifying an object of the subset of objects corresponding to the visual stimulus, stopping the display of the visual stimulus.

16. The method of claim 11 , wherein displaying a visual stimulus comprises altering the pixels representing one or more objects of the subset of objects to display a pattern including a solid color, a polychromatic spiral, or a checkerboard.

17. The method of claim 11 , wherein the frequency of one or more of the visual stimuli corresponds to a rate at which the pixels representing one or more objects of the subset of objects corresponding to the one or more visual stimuli alternate between a first color and a second color.

18. The method of claim 11 , wherein the frequency of one or more of the visual stimuli corresponds to a rate at which the one or more visual stimuli rotate.

19. The method of claim 11 , wherein determining the derived frequency comprises:

computing, for each template of a set of signal templates, a correlation between the visually evoked signal and the template, wherein each template corresponds to a distinct frequency; and

selecting a frequency corresponding to a template from the set of signal templates having a highest correlation as the derived frequency.

20. The method of claim 11 , wherein the frequency derived from the visually evoked signal is determined based on a plurality of signals measured at a plurality of electrodes on the wearable interface.

Assignments (2)
CHANGE OF NAME Recorded Nov 18, 2021
From: FACEBOOK, INC.
To: META PLATFORMS, INC.
Reel/Frame 058897/0824 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 9, 2019
From: WANG, YU-TE; CHEVILLET, MARK ALLAN
To: FACEBOOK, INC.
Reel/Frame 051215/0223 →
Cited By (2)
US 12,517,575 US 12,594,016