Neural network for eye image segmentation and image quality estimation
Systems and methods for eye image segmentation and image quality estimation are disclosed. In one aspect, after receiving an eye image, a device such as an augmented reality device can process the eye image using a convolutional neural network with a merged architecture to generate both a segmented eye image and a quality estimation of the eye image. The segmented eye image can include a background region, a sclera region, an iris region, or a pupil region. In another aspect, a convolutional neural network with a merged architecture can be trained for eye image segmentation and image quality estimation. In yet another aspect, the device can use the segmented eye image to determine eye contours such as a pupil contour and an iris contour. The device can use the eye contours to create a polar image of the iris region for computing an iris code or biometric authentication.
1. A method for training a convolutional neural network for eye image segmentation and image quality estimation, the method being performed by a system of one or more processors, and the method comprising:
obtaining a training set of eye images;
providing a convolutional neural network with the training set of eye images;
and training the convolutional neural network with the training set of eye images, wherein the convolution neural network comprises a segmentation tower and a quality estimation tower, wherein the segmentation tower comprises segmentation layers and shared layers,
wherein the quality estimation tower comprises quality estimation layers and the shared layers,
wherein an output layer of the shared layers is connected to a first input layer of the segmentation tower and a second input layer of the segmentation tower,
and wherein the output layer of the shared layers is connected to an input layer of the quality estimation layer.
2. The method of claim 1 , wherein training the convolutional neural network with the training set of eye images comprises:
processing an eye image of the training set using the segmentation tower to generate a segmentation of the eye image;
processing the eye image of the training set using the quality estimation tower to generate a quality estimation of the eye image;
computing a first difference between the segmentation of the eye image and a reference segmentation of the eye image; computing a second difference between the quality estimation of the eye image and a reference quality estimation of the eye image; and
updating parameters of the convolutional neural network using the first difference and the second difference.
3. The method of claim 2 , wherein updating the parameters of the convolutional neural network using the first difference and the second difference comprises setting a first percentage of the parameters of the convolutional neural network to values of zero during a first training cycle when training the convolutional neural network.
4. The method of claim 3 , wherein setting the first percentage of the parameters of the convolutional neural network to values of zero during the first training cycle when training the convolutional neural network comprises randomly setting the first percentage of the parameters of the convolutional neural network to values of zero during the first training cycle when training the convolutional neural network.
5. The method of claim 3 , wherein updating the parameters of the convolutional neural network using the first difference and the second difference further comprises setting a second percentage of the parameters of the convolutional neural network to values of zero during a second training cycle when training the convolutional neural network.
6. The method of claim 5 , wherein setting the second percentage of the parameters of the convolutional neural network to values of zero during the second training cycle when training the convolutional neural network comprises randomly setting the second percentage of the parameters of the convolutional neural network to values of zero during the second training cycle when training the convolutional neural network.
7. The method of claim 6 , wherein the first percentage or the second percentage is between 50% and 30%.
8. The method of claim 2 , wherein the segmentation of the eye image comprises a background, a sclera, an iris, or a pupil of the eye image, and wherein the reference segmentation of the eye image comprises a reference background, a reference sclera, a reference iris, or a reference pupil of the eye image.
9. The method of claim 1 , wherein the shared layers, the segmentation layers, or the quality estimation layers comprise a convolution layer, a brightness normalization layer, a batch normalization layer, a rectified linear layer, an upsampling layer, a concatenation layer, a pooling layer, a fully connected layer, a linear fully connected layer, a softsign layer, or any combination thereof.
10. The method of claim 1 , wherein an output of a first intermediate layer of the shared layers is connected to a third input layer of the segmentation layers.
11. A system comprising one or more processors and non-transitory computer storage media storing instructions that when executed by the one or more processors, cause the processors to perform operations comprising:
obtaining a training set of eye images;
providing a convolutional neural network with the training set of eye images;
and training the convolutional neural network with the training set of eye images, wherein the convolution neural network comprises a segmentation tower and a quality estimation tower, wherein the segmentation tower comprises segmentation layers and shared layers,
wherein the quality estimation tower comprises quality estimation layers and the shared layers,
wherein an output layer of the shared layers is connected to a first input layer of the segmentation tower and a second input layer of the segmentation tower,
and wherein the output layer of the shared layers is connected to an input layer of the quality estimation layer.
12. The system of claim 11 , wherein training the convolutional neural network with the training set of eye images comprises:
processing an eye image of the training set using the segmentation tower to generate a segmentation of the eye image;
processing the eye image of the training set using the quality estimation tower to generate a quality estimation of the eye image;
computing a first difference between the segmentation of the eye image and a reference segmentation of the eye image; computing a second difference between the quality estimation of the eye image and a reference quality estimation of the eye image; and
updating parameters of the convolutional neural network using the first difference and the second difference.
13. The system of claim 12 , wherein updating the parameters of the convolutional neural network using the first difference and the second difference comprises setting a first percentage of the parameters of the convolutional neural network to values of zero during a first training cycle when training the convolutional neural network.
14. The system of claim 13 , wherein setting the first percentage of the parameters of the convolutional neural network to values of zero during the first training cycle when training the convolutional neural network comprises randomly setting the first percentage of the parameters of the convolutional neural network to values of zero during the first training cycle when training the convolutional neural network.
15. The system of claim 13 , wherein updating the parameters of the convolutional neural network using the first difference and the second difference further comprises setting a second percentage of the parameters of the convolutional neural network to values of zero during a second training cycle when training the convolutional neural network.
16. The system of claim 12 , wherein the segmentation of the eye image comprises a background, a sclera, an iris, or a pupil of the eye image, and wherein the reference segmentation of the eye image comprises a reference background, a reference sclera, a reference iris, or a reference pupil of the eye image.
17. The system of claim 12 , wherein an output of a first intermediate layer of the shared layers is connected to a third input layer of the segmentation layers.
18. Non-transitory computer storage media storing instructions that when executed by a system of one or more processors, cause the one or more processors to perform operations comprising:
obtaining a training set of eye images;
providing a convolutional neural network with the training set of eye images;
and training the convolutional neural network with the training set of eye images, wherein the convolution neural network comprises a segmentation tower and a quality estimation tower, wherein the segmentation tower comprises segmentation layers and shared layers,
wherein the quality estimation tower comprises quality estimation layers and the shared layers,
wherein an output layer of the shared layers is connected to a first input layer of the segmentation tower and a second input layer of the segmentation tower,
and wherein the output layer of the shared layers is connected to an input layer of the quality estimation layer.
19. The computer storage media of claim 18 , wherein training the convolutional neural network with the training set of eye images comprises:
processing an eye image of the training set using the segmentation tower to generate a segmentation of the eye image;
processing the eye image of the training set using the quality estimation tower to generate a quality estimation of the eye image;
computing a first difference between the segmentation of the eye image and a reference segmentation of the eye image; computing a second difference between the quality estimation of the eye image and a reference quality estimation of the eye image; and
updating parameters of the convolutional neural network using the first difference and the second difference.
20. The computer storage media of claim 18 , wherein an output of a first intermediate layer of the shared layers is connected to a third input layer of the segmentation layers.