Neural network image processing apparatus

Inventors: Joseph Lemley (Galway, IE); Liviu-Cristian Dutu (Bucharest, RO); Stefan Mathe (Bucharest, RO); Madalin Dumitru-Guzu (Bucharest, RO); Dan Filip (Bucharest, RO)

G06V20/597 G06F3/013 G06F18/21 G06F18/214 G06N3/08 G06V10/82 G06V40/165 G06V40/166 G06V40/171 G06V40/19

View Patent ↗

Loading inventors, assignments & file history…

Monitor This Case

Get email alerts when status or documents change.

Order Certified Copies

Most orders are placed with the USPTO same day — all within 24 business hours.

Order via The Patent Place →

Pre-filled with this patent's details

Quick Facts

Patent No.: US 11,699,293
App. No.: 17/677,320
Granted: Jul 11, 2023
Kind: B2

External Links

USPTO Patent Center↗ Full Patent PDF↗

Abstract

A neural network image processing apparatus arranged to acquire images from an image sensor and to: identify a ROI containing a face region in an image; determine at plurality of facial landmarks in the face region; use the facial landmarks to transform the face region within the ROI into a face region having a given pose; and use transformed landmarks within the transformed face region to identify a pair of eye regions within the transformed face region. Each identified eye region is fed to a respective first and second convolutional neural network, each network configured to produce a respective feature vector. Each feature vector is fed to respective eyelid opening level neural networks to obtain respective measures of eyelid opening for each eye region. The feature vectors are combined and to a gaze angle neural network to generate gaze yaw and pitch values substantially simultaneously with the eyelid opening values.

Claims (72)

1. A method comprising:

identifying a face region in an image;

determining a plurality of facial landmarks in the face region;

determining, based at least in part on the plurality of facial landmarks, a pose of the face region;

identifying, based at least in part on the pose, a first eye region and a second eye region within the face region;

inputting the first eye region into a first neural network and the second eye region into a second neural network;

receiving a first feature vector from the first neural network and a second feature vector from the second neural network;

determining a first eyelid opening value based at least in part on the first feature vector and a second eyelid opening value based at least in part on the second feature vector;

inputting the first eyelid opening value and the second eyelid opening value into a third neural network; and

receiving, from the third neural network, a gaze yaw value or a pitch value associated with the first eyelid opening value or the second eyelid opening value.

2. The method of claim 1 , wherein the first neural network, the second neural network, and the third neural network are associated with a common training set.

3. The method of claim 1 , wherein:

the first neural network is a first convolutional neural network, and the second neural network is a second convolutional neural network.

4. The method of claim 3 , wherein first convolutional neural network or the second convolutional neural network comprises a convolutional layer and a pooling layer.

5. The method of claim 1 , wherein the third neural network comprises fully connected layers.

6. The method of claim 1 , further comprising:

inputting the first feature vector into a fourth neural network;

inputting the second feature vector into a fifth neural network;

determining the first eyelid opening value based at least in part on a first output from the fourth neural network; and

determining the second eyelid opening value based at least in part on a second output from the fifth neural network.

7. The method of claim 1 , further comprising:

processing a first layer associated with the first neural network and a second layer associated with the second neural network substantially simultaneously.

8. The method of claim 1 , wherein the first feature vector or the second feature vector comprises a plurality of floating point numbers.

9. The method of claim 1 , wherein the plurality of facial landmarks in the face region comprises two or more of: a first eye, a second eye, a first eyebrow, a second eyebrow, a jaw, a mouth, or a nose.

10. The method of claim 1 , wherein the plurality of facial landmarks in the face region are associated with a facial feature other than an eye.

11. The method of claim 1 , wherein the first eye region and the second eye region comprise a same aspect ratio.

12. The method of claim 1 , further comprising:

determining face information encoding at least one of: a pose or an orientation for a face in the face region;

combining, as a third feature vector, the face information with the first feature vector or the second feature vector; and

determining the first eyelid opening value or the second eyelid opening value based at least in part on the third feature vector.

13. The method of claim 1 , further comprising determining a location of a pupil within the first eye region or the second eye region based at least in part on illumination from a light source.

14. The method of claim 13 , further comprising:

determining an intensity of the light source to illuminate the pupil; and

reducing a saturation region associated with the pupil based at least in part on the intensity of the light source.

15. The method of claim 13 , wherein the image is a first image, and further comprising:

receiving a second image;

determining a change in the first eye region or the second eye region between the first image and the second image; and

inputting the change in the first eye region or the second eye region into the first neural or the second neural network.

16. One or more non-transitory computer-readable media storing instructions executable by a processor, wherein the instructions, when executed, cause the processor to perform operations comprising: