Neural network image processing apparatus
A neural network image processing apparatus arranged to acquire images from an image sensor and to: identify a ROI containing a face region in an image; determine at plurality of facial landmarks in the face region; use the facial landmarks to transform the face region within the ROI into a face region having a given pose; and use transformed landmarks within the transformed face region to identify a pair of eye regions within the transformed face region. Each identified eye region is fed to a respective first and second convolutional neural network, each network configured to produce a respective feature vector. Each feature vector is fed to respective eyelid opening level neural networks to obtain respective measures of eyelid opening for each eye region. The feature vectors are combined and to a gaze angle neural network to generate gaze yaw and pitch values substantially simultaneously with the eyelid opening values.
1. An apparatus comprising:
one or more processors configured to:
acquire an image from an image sensor;
identify a region of interest containing a face region in the image;
determine a plurality of facial landmarks in the face region within the region of interest;
use said plurality of said facial landmarks to transform said face region into a transformed face region having a given pose;
use transformed landmarks within said transformed face region to identify a pair of eye regions within said transformed face region;
feed each identified eye region of said pair of eye regions to a respective first and second convolutional neural network, each network configured to produce a respective feature vector comprising a plurality of numerical values;
feed each feature vector to respective eyelid opening level neural networks to obtain a respective eyelid opening value for each eye region;
combine the feature vectors into a combined feature vector; and
feed the combined feature vector to a gaze angle neural network to generate gaze yaw and pitch values substantially simultaneously with the eyelid opening values, wherein said eyelid opening level neural networks and said gaze angle neural network are jointly trained based on a common training set.
2. The apparatus according to claim 1 further comprising a multi-processor neural network core which substantially simultaneously processes layers from said first and second convolutional networks and which substantially simultaneously processes layers from said eyelid opening level neural networks and said gaze angle neural network.
3. The apparatus according to claim 1 wherein at least one of said feature vectors comprises a plurality of floating point numbers.
4. The apparatus according to claim 1 wherein at least one of said feature vectors comprises at least 128 32-bit floating point numbers.
5. The apparatus according to claim 1 wherein said given pose is a front facing pose.
6. The apparatus according to claim 1 wherein at least one of said pair of eye regions comprises an area including an eye related landmark or an eyebrow related landmark.
7. The apparatus according to claim 1 wherein at least one of said pair of eye regions is rectangular.
8. The apparatus according to claim 1 wherein at least one of said pair of eye regions comprises a fixed aspect ratio.
9. The apparatus according to claim 1 wherein at least one of said first convolutional neural network or said second convolutional neural network comprises a combination of a convolutional layer and a pooling layer.
10. The apparatus according to claim 9 wherein said convolution layer comprises an activation function.
11. The apparatus according to claim 1 wherein said gaze angle network comprises a fully connected layer.
12. The apparatus according to claim 1 wherein at least one of said eyelid opening level neural networks comprises a fully connected layer.
13. The apparatus according to claim 1 wherein said facial landmarks delineate at least one of a jaw, mouth, nose, eye, or eyebrow.
14. The apparatus according to claim 1 wherein said facial landmarks do not delineate any feature on an eye.
15. The apparatus according to claim 1 wherein said apparatus is further configured to combine said feature vectors by concatenating the feature vectors.
16. The apparatus according to claim 1 wherein said feature vectors comprise either 1×M or A×B=M values, wherein A, B, and M are integers.
17. A system comprising:
one or more processors;
memory communicatively coupled to the one or more processors, the memory storing instructions that, when executed by the one or more processors, causes the system to perform operations comprising:
acquiring an image from an image sensor;
identifying a region of interest containing a face region in the image;
determining a plurality of facial landmarks in the face region within the region of interest;
using said plurality of facial landmarks to transform said face region into a transformed face region having a given pose;
using transformed landmarks within said transformed face region to identify a pair of eye regions within said transformed face region;
feeding each identified eye region of said pair of eye regions to a respective first and second convolutional neural network, each network configured to produce a respective feature vector comprising a plurality of numerical values;
feeding each feature vector to respective eyelid opening level neural networks to obtain a respective eyelid opening value for each eye region;
combining the feature vectors into a combined feature vector; and
feeding the combined feature vector to a gaze angle neural network to generate gaze yaw and pitch values substantially simultaneously with the eyelid opening values, wherein said eyelid opening level neural networks and said gaze angle neural network are jointly trained based on a common training set.
18. The system according to claim 17 , wherein at least one of said pair of eye regions comprises an area including an eye related landmark or an eyebrow related landmark.
19. A method comprising:
acquiring an image from an image sensor;
identifying a region of interest containing a face region in the image;
determining a plurality of facial landmarks in the face region within the region of interest;
using said plurality of facial landmarks to transform said face region into a transformed face region having a given pose;
using transformed landmarks within said transformed face region to identify a pair of eye regions within said transformed face region;
feeding each identified eye region of said pair of eye regions to a respective first and second convolutional neural network, each network configured to produce a respective feature vector comprising a plurality of numerical values;
feeding each feature vector to respective eyelid opening level neural networks to obtain a respective eyelid opening value for each eye region;
combining the feature vectors into a combined feature vector; and
feeding the combined feature vector to a gaze angle neural network to generate gaze yaw and pitch values substantially simultaneously with the eyelid opening values, wherein said eyelid opening level neural networks and said gaze angle neural network are jointly trained based on a common training set.
20. The method according to claim 19 , wherein at least one of said pair of eye regions comprises an area including an eye related landmark or an eyebrow related landmark.