IP Library Granted Patent US 9,996,743
Granted Patent B2
US 9,996,743 · App. 14/646,632 · Granted Jun 12, 2018

Methods, systems, and media for detecting gaze locking

Inventors: Brian Anthony Smith (Brentwood, NY); Qi Yin (New York, NY); Shree Kumar Nayar (New York, NY)
Assignee: The Trustees of Columbia University in the City of New York
G06K9/00604G06K9/0061G06K9/00248G06K9/4604G06K9/4671G06T7/73G06T2207/30201
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 9,996,743
App. No.
14/646,632
Granted
Jun 12, 2018
Kind
B2
Abstract

Methods, systems, and media for detecting gaze locking are provided. In some embodiments, methods for gaze locking are provided, the methods comprising: receiving an input image including a face; locating a pair of eyes in the face of the input image; generating a coordinate frame based on the pair of eyes; identifying an eye region in the coordinate frame; generating, using a hardware processor, a feature vector based on values of pixels in the eye region; and determining whether the face is gaze locking based on the feature vector.

Claims (73)

1. A method for detecting gaze locking, the method comprising:

receiving an input image including a face;

locating a pair of eyes in the face of the input image;

generating a first coordinate frame based on one of the pair of eyes and generating a second coordinate frame based on another of the pair of eyes;

concatenating the first coordinate frame and the second coordinate frame to produce a concatenated coordinate frame;

identifying an eye region in the concatenated coordinate frame;

generating, using a hardware processor, a feature vector by concatenating values of pixels in the eye region; and

determining, based on the feature vector, whether the face is gaze locking.

2. The method of claim 1 , further comprising:

identifying a set of fiducial points in the input image; and

locating the pair of eyes in the input image based on the set of fiducial points.

3. The method of claim 2 , further comprising identifying the eye region in the concatenated coordinate frame based on the set of fiducial points.

4. The method of claim 1 , further comprising applying a first affine transformation to a portion of the input image corresponding to the pair of eyes to produce the first coordinate frame.

5. The method of claim 1 , further comprising concatenating intensity values of the pixels in the eye region to produce the feature vector.

6. The method of claim 1 , further comprising applying a mask to the concatenated coordinate frame to produce a masked image.

7. The method of claim 6 , further comprising identifying the eye region based on the masked image.

8. The method of claim 6 , further comprising:

concatenating intensity values of pixels in the masked image to produce the feature vector;

normalizing the feature vector to produce a normalized feature vector; and

determining whether the face is gaze locking based on the normalized feature vector.

9. The method of claim 1 , further comprising:

compressing the feature vector to generate a compressed feature vector; and

determining, based on the compressed feature vector, whether the face is gaze locking using a classifier.

10. The method of claim 9 , wherein the classifier is a binary classifier.

11. A system for detecting gaze locking, the system comprising:

at least one hardware processor that is configured to:

receive an input image including a face;

locate a pair of eyes in the face of the input image;

generate a first coordinate frame based on one of the pair of eyes and generate a second coordinate frame based on another of the pair of eyes;

concatenate the first coordinate frame and the second coordinate frame to produce a concatenated coordinate frame;

identify an eye region in the concatenated coordinate frame;

generate a feature vector by concatenating values of pixels in the eye region; and

determine, based on the feature vector, whether the face is gaze locking.

12. The system of claim 11 , wherein the hardware processor is further configured to:

identify a set of fiducial points in the input image; and

locate the pair of eyes in the input image based on the set of fiducial points.

13. The system of claim 12 , wherein the hardware processor is further configured to identify the eye region in the concatenated coordinate frame based on the set of fiducial points.

14. The system of claim 11 , wherein the hardware processor is further configured to apply a first affine transformation to a portion of the input image corresponding to the pair of eyes to produce the first coordinate frame.

15. The system of claim 11 , wherein the hardware processor is further configured to concatenate intensity values of the pixels in the eye region to produce the feature vector.

16. The system of claim 11 , wherein the hardware processor is further configured to apply a mask to the concatenated coordinate frame to produce a masked image.

17. The system of claim 16 , wherein the hardware processor is further configured to identify the eye region based on the masked image.

18. The system of claim 16 , wherein the hardware processor is further configured to:

concatenate intensity values of pixels in the masked image to produce the feature vector;

normalize the feature vector to produce a normalized feature vector; and

determine whether the face is gaze locking based on the normalized feature vector.

19. The system of claim 11 , wherein the hardware processor is further configured to:

compress the feature vector to generate a compressed feature vector; and

determine, based on the compressed feature vector, whether the face is gaze locking using a classifier.

20. The system of claim 19 , wherein the classifier is a binary classifier.

21. A non-transitory computer-readable medium containing computer-executable instructions that, when executed by a processor, cause the processor to perform a method for detecting gaze locking, the method comprising:

receiving an input image including a face;

locating a pair of eyes in the face of the input image;

generating a first coordinate frame based on one of the pair of eyes and generating a second coordinate frame based on another of the pair of eyes;

concatenating the first coordinate frame and the second coordinate frame to produce a concatenated coordinate frame;

identifying an eye region in the concatenated coordinate frame;

generating a feature vector by concatenating values of pixels in the eye region; and

determining, based on the feature vector, whether the face is gaze locking.

22. The non-transitory computer-readable medium of claim 21 , wherein the method further comprises:

identifying a set of fiducial points in the input image; and

locating the pair of eyes in the input image based on the set of fiducial points.

23. The non-transitory computer-readable medium of claim 22 , wherein the method further comprises identifying the eye region in the concatenated coordinate frame based on the set of fiducial points.

24. The non-transitory computer-readable medium of claim 21 , wherein the method further comprises applying a first affine transformation to a portion of the input image corresponding to the pair of eyes to produce the first coordinate frame.

25. The non-transitory computer-readable medium of claim 21 , wherein the method further comprises concatenating intensity values of the pixels in the eye region to produce the feature vector.

26. The non-transitory computer-readable medium of claim 21 , wherein the method further comprises applying a mask to the concatenated coordinate frame to produce a masked image.

27. The non-transitory computer-readable medium of claim 26 , wherein the method further comprises identifying the eye region based on the masked image.

28. The non-transitory computer-readable medium of claim 26 , wherein the method further comprises:

concatenating intensity values of pixels in the masked image to produce the feature vector;

normalizing the feature vector to produce a normalized feature vector; and

determining whether the face is gaze locking based on the normalized feature vector.

29. The non-transitory computer-readable medium of claim 21 , wherein the method further comprises:

compressing the feature vector to generate a compressed feature vector; and

determining, based on the compressed feature vector, whether the face is gaze locking using a classifier.

30. The non-transitory computer-readable medium of claim 29 , wherein the classifier is a binary classifier.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Feb 24, 2022
From: SMITH, BRIAN ANTHONY; NAYAR, SHREE KUMAR
To: THE TRUSTEES OF COLUMBIA UNIVERSITY IN THE CITY OF NEW YORK
Reel/Frame 059094/0905 →
CONFIRMATORY LICENSE Recorded Apr 9, 2018
From: COLUMBIA UNIVERSITY
To: NAVY, SECRETARY OF THE UNITED STATES OF AMERICA
Reel/Frame 045877/0784 →
Continuity (2)
Provisional Application 61730933 · Nov 28, 2012
Related Publication 20150302251A1 · Oct 22, 2015