IP Library Granted Patent US 9,471,763
Granted Patent B2
US 9,471,763 · App. 13/464,703 · Granted Oct 18, 2016

User input processing with eye tracking

Inventor: Chris Norden (Austin, TX)
Assignee: SONY INTERACTIVE ENTERTAINMENT AMERICA LLC
G06F21/32G06F3/013G06F3/0488G06F2221/2133G06K9/00597
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 9,471,763
App. No.
13/464,703
Granted
Oct 18, 2016
Kind
B2
Abstract

A system determines which user of multiple users provided input through a single input device. A mechanism captures images of the one or more users. When input is detected, the images may be processed to determine which user provided an input using the input device. The images may be processed to identify each users head and eyes, and determine the focus point for each user's eyes. The user which has eyes focused at the input location is identified as providing the input. When the input mechanism is a touch screen, the user having eyes focused on the touch screen portion which was touched is identified as the source of the input.

Claims (44)

1. A method for receiving input at a touch-sensitive device, comprising:

capturing image data for a physical area surrounding the touch-sensitive device;

identifying a plurality of physically present users at the touch-sensitive device by identifying head candidates from the captured image data, wherein the identifying includes:

analyzing the captured image data for shapes resembling a human head, wherein the shapes are detected using contrast detection and motion detection,

evaluating each shape in the captured image data resembling a human head for the presence of features common to most human heads, wherein the features include contrast or shading present on the shape based on where the nose, mouth or eye may be found,

assigning the evaluated shape a value corresponding to a likelihood that the evaluated shape is a human head based on the presence of one or more features, and

determining that the assigned value for a particular evaluated shape is greater than a threshold indicative that the evaluated shape corresponds to a participating user;

identifying facial features for each of the physically present users, the facial features including the eyes of each of the plurality of physically present users;

calibrating the eyes of each of the plurality of physically present users prior to engaging in real-time interactions with the touch-sensitive device, the calibration including focus with a plurality of pre-defined regions of the touch-sensitive device;

tracking an eye focus for the eyes of each of the plurality of physically present users in real-time and following calibration, the eye focus including an analysis of the area and location of sclera versus pupil for each eye of each of the plurality of physically present users;

receiving a touch input at the touch-sensitive device, wherein the touch input is detected at a hot spot on a touch screen display of the touch-sensitive device; and

associating the touch input with an acting user of the plurality of physically present users based on the tracking of the eye focus of each of the plurality of physically present users, wherein the acting user is determined to have an eye focus closest to the hot spot as a result of the analysis of the area and location of sclera versus pupil for the eyes of each of the plurality of physically present users.

2. The method of claim 1 , wherein the identifying of the facial features of each of the physically present users includes identifying contours of a face of the user, the contours used to identify a particular face of a user.

3. The method of claim 1 , further comprising evaluating distances of each of the physically present users from the touch-sensitive device, wherein users being a distance greater than a pre-determined threshold from the touch-sensitive device are determined as not capable of providing user input.

4. The method of claim 1 , wherein the identifying is performed with a feature library, the feature library including facial and eye masks, templates, and models used to process an image, identify a phsyical feature and identify the state of the physical feature.

5. The method of claim 4 , wherein the state of the physical feature includes a direction where the eyes of the user are focused at.

6. The method of claim 1 , wherein aassociating the input with an acting user further utilizes additional considerations including the likelihood of receiving input from each user based on input history, current eye focus, and whether an input is expected from a particlar user.

7. A non-transitory computer-readable storage medium having embodied thereon a program, the program being executable by a processor to perform a method for receiving input at a touch-sensitive device, the method comprising:

capturing image data for a physical area surrounding the touch-sensitive device;

identifying a plurality of physically present users at the touch-sensitive device by identifying head candidates from the captured image data, wherein the identifying includes:

analyzing the captured image data for shapes resembling a human head, wherein the shapes are detected using contrast detection and motion detection,

evaluating each shape in the captured image data resembling a human head for the presence of features common to most human heads, wherein the features include contrast or shading present on the shape based on where the nose, mouth or eye may be found,

assigning the evaluated shape a value corresponding to a likelihood that the evaluated shape is a human head based on the presence of one or more features, and

determining that the assigned value for a particular evaluated shape is greater than a threshold indicative that the evaluated shape corresponds to a participating user;

identifying facial features for each of the physically present users, the facial features including the eyes of each of the plurality of physically present users;

calibrating the eyes of each of the plurality of physically present users prior to engaging in real-time interactions with the touch-sensitive device, the calibration including focus with a plurality of pre-defined regions of the touch-sensitive device;

tracking an eye focus for the eyes of each of the plurality of physically present users in real-time and following calibration, the eye focus including an analysis of the area and location of sclera versus pupil for each eye of each of the plurality of physically present users;

receiving a touch input at the touch-sensitive device, wherein the touch input is detected at a hot spot on a touch screen display of the touch-sensitive device; and

associating the touch input with an acting user of the plurality of physically present users based on the tracking of the eye focus of each of the plurality of physically present users, wherein the acting user is determined to have an eye focus closest to the hot spot as a result of the analysis of the area and location of sclera versus pupil for the eyes of each of the plurality of physically present users.

8. A system for detecting input, the system comprising:

a touch-sensitive display device that receives a touch input at a hot spot on a touch screen display of the touch-sensitive device;

a camera that captures color image data for a physical area surrounding the touch-sensitive display device;

a non-volatile storage device coupled to the camera to receive the captured image data;

a processor that executes non-transitory computer-readable instructions stored in memory, wherein execution of the instructions:

identifies a plurality of physically present users at the touch-sensitive device by identifying head candidates from the captured image data maintained in the non-volatile storage device, wherein the identifying includes:

analyzing the captured image data for shapes resembling a human head, wherein the shapes are detected using contrast detection and motion detection,

evaluating each shape in the captured image data resembling a human head for the presence of features common to most human heads, wherein the features include contrast or shading present on the shape based on where the nose, mouth or eye may be found,

assigning the evaluated shape a value corresponding to a likelihood that the evaluated shape is a human head based on the presence of one or more features, and

determining that the assigned value for a particular evaluated shape is greater than a threshold indicative that the evaluated shape corresponds to a participating user,

identifies facial features for each of the physically present users, the facial features including the eyes of each of the plurality of physically present users,

calibrates the eyes of each of the plurality of physically present users prior to engaging in real-time interactions with the touch-sensitive device, the calibration including focus with a plurality of pre-defined regions of the touch-sensitive device,

tracks an eye focus for the eyes of each of the plurality of physically present users in real-time and following calibration, the eye focus including an analysis of the area and location of sclera versus pupil for each eye of each of the plurality of physically present users, and

associating the received touch input on the touch-screen display with an acting user of the plurality of physically present users based on the tracking of the eye focus of each of the plurality of physically present users, wherein the acting user is determined to have an eye focus closest to the hot spot as a result of the analysis of the area and location of sclera versus pupil for the eyes of each of the plurality of physically present users.

9. The system of claim 8 , further comprising an infra-red (IR) device that captures IR images of one or more users of the plurality of physically present users, the captured IR images used in conjunction with the camera to identify head candidates.

Assignments (4)
MERGER Recorded Mar 30, 2020
From: SONY INTERACTIVE ENTERTAINMENT AMERICA LLC
To: SONY INTERACTIVE ENTERTAINMENT LLC
Reel/Frame 053323/0567 →
CHANGE OF NAME Recorded May 5, 2016
From: SONY COMPUTER ENTERTAINMENT AMERICA LLC
To: SONY INTERACTIVE ENTERTAINMENT AMERICA LLC
Reel/Frame 038626/0637 →
CORRECTIVE ASSIGNMENT TO CORRECT THE ASSIGNEE PREVIOUSLY RECORDED ON REEL 028555 FRAME 0410. ASSIGNOR(S) HEREBY CONFIRMS THE ASSIGNEE SHOULD READ SONY COMPUTER ENTERTAINMENT AMERICA LLC. Recorded Jul 19, 2012
From: NORDEN, CHRIS
To: SONY COMPUTER ENTERTAINMENT AMERICA LLC
Reel/Frame 028598/0059 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jul 13, 2012
From: NORDEN, CHRIS
To: SONY COMPUTER ENTERTAINMENT AMERICA INC.
Reel/Frame 028555/0410 →
Continuity (1)
Related Publication 20130293467A1 · Nov 7, 2013