Neural network for eye image segmentation and image quality estimation
Systems and methods for eye image segmentation and image quality estimation are disclosed. In one aspect, after receiving an eye image, a device such as an augmented reality device can process the eye image using a convolutional neural network with a merged architecture to generate both a segmented eye image and a quality estimation of the eye image. The segmented eye image can include a background region, a sclera region, an iris region, or a pupil region. In another aspect, a convolutional neural network with a merged architecture can be trained for eye image segmentation and image quality estimation. In yet another aspect, the device can use the segmented eye image to determine eye contours such as a pupil contour and an iris contour. The device can use the eye contours to create a polar image of the iris region for computing an iris code or biometric authentication.
1. A system for eye image segmentation and image quality estimation, the system comprising:
an eye-imaging camera configured to obtain an eye image;
non-transitory memory configured to store the eye image;
a hardware processor in communication with the non-transitory memory, the hardware processor programmed to:
receive the eye image;
process the eye image using a convolution neural network to generate a segmentation of the eye image; and
process the eye image using the convolution neural network to generate a quality estimation of the eye image,
wherein the convolution neural network comprises a segmentation tower and a quality estimation tower,
wherein the segmentation tower comprises segmentation layers and shared layers,
wherein the quality estimation tower comprises quality estimation layers and the shared layers,
wherein a first output layer of the shared layers is connected to a first input layer of the segmentation tower and to a second input layer of the segmentation tower, at least one of the first input layer or the second input layer comprising a concatenation layer,
wherein the first output layer of the shared layers is connected to an input layer of the quality estimation layer, and
wherein the eye image is received by an input layer of the shared layers.
2. The system of claim 1 , wherein a second output layer of the shared layers is connected to a third input layer of the segmentation tower, the third input layer comprising a concatenation layer.
3. The system of claim 1 , wherein to process the eye image using the convolution neural network to generate the segmentation of the eye image, the hardware processor is programmed to
generate the segmentation of the eye image using the segmentation tower,
wherein an output of an output layer of the segmentation tower comprises the segmentation of the eye image.
4. The system of claim 3 , wherein the segmentation of the eye image includes a background, a sclera, an iris, or a pupil of the eye image.
5. The system of claim 4 , wherein the hardware processor is further programmed to:
determine a pupil contour of an eye in the eye image using the segmentation of the eye image;
determine an iris contour of the eye in the eye image using the segmentation of the eye image; and
determine a mask for an irrelevant area in the eye image.
6. The system of claim 1 , wherein the shared layers are configured to encode the eye image by decreasing a spatial dimension of feature maps and increasing a number of feature maps computed by the shared layers.
7. The system of claim 6 , wherein the segmentation layers are configured to decode the eye image encoded by the shared layers by increasing the spatial dimension of the feature maps and reducing the number of feature maps.
8. The system of claim 1 , wherein to process the eye image using the convolution neural network to generate the quality estimation of the eye image, the hardware processor is programmed to:
generate the quality estimation of the eye image using the quality estimation tower,
wherein an output of an output layer of the quality estimation tower comprises the quality estimation of the eye image.
9. The system of claim 1 , wherein the quality estimation tower is configured to output at least two channels of output, wherein a first of the at least two channels comprises a good quality estimation and a second of the at least two channels comprises a bad quality estimation.
10. The system of claim 1 , wherein the shared layers, the segmentation layers, or the quality estimation layers comprise a convolution layer, a brightness normalization layer, a batch normalization layer, a rectified linear layer, an upsampling layer, a concatenation layer, a pooling layer, a fully connected layer, a linear fully connected layer, a softsign layer, or any combination thereof.
11. A system for eye image segmentation and image quality estimation, the system comprising:
an eye-imaging camera configured to obtain an eye image;
non-transitory memory configured to store the eye image;
a hardware processor in communication with the non-transitory memory, the hardware processor programmed to:
receive the eye image;
process the eye image using a convolution neural network to generate a segmentation of the eye image; and
process the eye image using the convolution neural network to generate a quality estimation of the eye image,
wherein the convolution neural network comprises a segmentation tower and a quality estimation tower,
wherein the segmentation tower comprises segmentation layers and shared layers,
wherein the quality estimation tower comprises quality estimation layers and the shared layers,
wherein the segmentation layers are not shared with the quality estimation tower,
wherein the quality estimation layers are not shared with the segmentation tower, and
wherein the eye image is received by an input layer of the shared layers.
12. The system of claim 11 , wherein a first output layer of the shared layers is connected to a first input layer of the segmentation tower.
13. The system of claim 12 , wherein the first output layer of the shared layers is connected to a second input layer of the segmentation tower,
wherein the first input layer or the second input layer comprises a concatenation layer.
14. The system of claim 12 , wherein the first output layer of the shared layers is further connected to an input layer of the quality estimation tower.
15. The system of claim 11 ,
wherein to process the eye image using the convolution neural network to generate the segmentation of the eye image, the hardware processor is programmed to:
generate the segmentation of the eye image using the segmentation tower,
wherein an output of an output layer of the segmentation tower comprises the segmentation of the eye image.
16. The system of claim 11 , wherein the segmentation of the eye image includes a background, a sclera, an iris, or a pupil of the eye image.
17. The system of claim 11 , wherein to process the eye image using the convolution neural network to generate the quality estimation of the eye image, the hardware processor is programmed to:
generate the quality estimation of the eye image using the quality estimation tower,
wherein an output of an output layer of the quality estimation tower comprises the quality estimation of the eye image.
18. The system of claim 11 , wherein the shared layers, the segmentation layers, or the quality estimation layers comprise a convolution layer, a batch normalization layer, a rectified linear layer, an upsampling layer, a concatenation layer, a pooling layer, a fully connected layer, a linear fully connected layer, or any combination thereof.
19. The system of claim 18 , wherein the batch normalization layer is a batch local contrast normalization layer or a batch local response normalization layer.
20. The system of claim 11 , wherein the shared layers, the segmentation layers, or the quality estimation layers comprise a brightness normalization layer, a softsign layer, or any combination thereof.