IP Library › Granted Patent US 11,875,603
Granted Patent B2
US 11,875,603 · App. 17/425,769 · Granted Jan 16, 2024

Facial action unit detection

Inventors: Shaoyuan Xu (Palo Alto, CA); Qian Lin (Palo Alto, CA); Jan Allebach (Boise, ID)
Assignees: Hewlett-Packard Development Company, L.P.; Purdue Research Foundation
G06V40/175G06V10/70G06V10/761G06V40/161G06V40/165G06V40/171G06V40/176G06V40/18
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,875,603
App. No.
17/425,769
Granted
Jan 16, 2024
Kind
B2
Abstract

An example system includes a landmark engine to detect a facial landmark in an image of a face. The system includes a comparison engine to determine a difference between the facial landmark in the image and a facial landmark of a neutral face. The system also includes an action engine to determine whether a facial action unit occurred based on whether the difference satisfies a condition.

Claims (30)

1. A system comprising:

a landmark engine to determine a facial landmark for an image of a face;

a comparison engine to determine a difference between the facial landmark for the image and a facial landmark of a neutral face; and

an action engine to determine whether a facial action unit occurred based on whether the difference satisfies a condition,

wherein the action engine includes a machine-learning model to determine at least one of an emotion, an amount of pain, a mental condition, or a truthfulness of the target face based on the difference.

2. The system of claim 1 , further comprising an alignment engine to generate the image of the face by cropping an initial image of the face, wherein the alignment engine is to select a size and an alignment of the face in the image based on a size and an alignment of the neutral face.

3. The system of claim 1 , wherein the action engine is to determine whether each of a plurality of facial action units occurred, and further comprising an expression engine to determine an emotion of the face based on whether the plurality of facial action units occurred.

4. The system of claim 1 , wherein the difference is a relative difference, and wherein the comparison engine is to determine the relative difference based on a raw difference and a reference distance from the neutral face or the face.

5. A method, comprising:

capturing an image of a target face;

selecting an image of a neutral face corresponding to the target face;

determining a plurality of facial landmarks in the image of the target face and a plurality of facial landmarks in the image of the neutral face;

selecting a facial action unit;

determining a difference between a selected landmark from the plurality of facial landmarks in the image of the target face and a corresponding landmark from the plurality of landmarks in the image of the neutral face, wherein movement of the selected landmark is associated with the facial action unit;

determining that the facial action unit occurred based on the difference satisfying a condition, and

determining an expression of the target face based on the determining that the facial action unit occurred,

wherein determining the expression of the target face comprises determining at least one of an emotion, an amount of pain, a mental condition, or a truthfulness of the target face.

6. The method of claim 5 , further comprising detecting the target face in the image of the target face, wherein determining the plurality of facial landmarks in the image of the target face comprises determining the plurality of facial landmarks based on the detecting of the target face.

7. The method of claim 5 , wherein determining the facial action unit occurred comprises determining the difference exceeds a threshold.

8. A non-transitory computer-readable medium comprising instructions that, when executed by a processor, cause the processor to:

determine a first distance in an image of a target face;

modify the image of the target face based on the first distance to create a modified image of the target face;

determine a difference between a landmark in the modified image of the target face and a corresponding landmark in an image of a neutral face;

determine a second distance based on the image of the neutral face or the modified image of the target face; and

determine an expression of the target face based on the difference relative to the second distance,

wherein determining the expression of the target face comprises determining at least one of an emotion, an amount of pain, a mental condition, or a truthfulness of the target face.

9. The computer-readable medium of claim 8 , the instructions to cause the processor to determine the expression include instructions to cause the processor to determine whether a facial action unit occurred based on whether the difference normalized by the second distance exceeds a threshold.

10. The computer-readable medium of claim 8 , wherein the first distance equals the second distance.

11. The computer-readable medium of claim 8 , wherein the instructions cause the processor to determine the first distance based on a distance between eye centers and to determine the second distance based on landmarks at outer edges of eyes.

12. The computer-readable medium of claim 11 , wherein the instructions to cause the processor to determine the distance between the eye centers include instructions to cause the processor to determine a plurality of landmarks around an eye and average the plurality of landmarks to determine an eye center for the eye.

Assignments (3)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Nov 10, 2023
From: ALLEBACH, JAN P; XU, SHAOYUAN
To: PURDUE RESEARCH FOUNDATION
Reel/Frame 065520/0765 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Oct 13, 2023
From: ALLEBACH, JAN P.; XU, SHAOYUAN
To: PURDUE RESEARCH FOUNDATION
Reel/Frame 065213/0079 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jul 26, 2021
From: LIN, QIAN
To: HEWLETT-PACKARD DEVELOPMENT COMPANY, L.P.
Reel/Frame 056974/0696 →
Continuity (1)
Related Publication 20220114838A1 · Apr 14, 2022