IP Library › Granted Patent US 12,462,394
Granted Patent B2
US 12,462,394 · App. 18/455,093 · Granted Nov 4, 2025

Neural network for eye image segmentation and image quality estimation

Inventors: Alexey Spizhevoy (Novgorod, RU); Adrian Kaehler (Los Angeles, CA); Vijay Badrinarayanan (Los Altos, CA)
Assignee: Magic Leap, Inc.
G06T7/12G06F18/2413G06T7/0002G06T7/10G06T7/11G06T7/194G06V10/454G06V10/56G06V10/764G06V10/82G06V10/993G06V40/193G06V40/197G06T2207/10024G06T2207/20076G06T2207/20081G06T2207/20084G06T2207/30041G06T2207/30168G06T2207/30196
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,462,394
App. No.
18/455,093
Granted
Nov 4, 2025
Kind
B2
Abstract

Systems and methods for eye image segmentation and image quality estimation are disclosed. In one aspect, after receiving an eye image, a device such as an augmented reality device can process the eye image using a convolutional neural network with a merged architecture to generate both a segmented eye image and a quality estimation of the eye image. The segmented eye image can include a background region, a sclera region, an iris region, or a pupil region. In another aspect, a convolutional neural network with a merged architecture can be trained for eye image segmentation and image quality estimation. In yet another aspect, the device can use the segmented eye image to determine eye contours such as a pupil contour and an iris contour. The device can use the eye contours to create a polar image of the iris region for computing an iris code or biometric authentication.

Claims (72)

1 . A computer-implemented method for determining eye contours in a semantically segmented eye image, comprising:

receiving a semantically segmented eye image of an eye image comprising a plurality of pixels;

determining a pupil contour using the semantically segmented eye image;

determining an iris contour using the semantically segmented eye image,

wherein,

the pupil contour is determined using a first binary image created based on the semantically segmented eye image, wherein a color value of a first binary image pixel of the first binary image is a fourth color value or a third color value, and/or

the iris contour is determined using a second binary image created based on the semantically segmented eye image, wherein a color value of a second binary image pixel of the second binary image is the third color value or a second color value; and

performing personal biometric identification based the pupil contour and the iris contour.

2 . The computer-implemented method of claim 1 , wherein a first color value is greater than the second color value, wherein the second color value is greater than the third color value, and wherein the third color value is greater than the fourth color value.

3 . The computer-implemented method of claim 1 , wherein determining the pupil contour using the semantically segmented eye image, comprises:

determining a pupil contour border;

removing a plurality of pixels from the pupil contour border; and

determining the pupil contour as an ellipse from remaining pixels of the pupil contour border.

4 . The computer-implemented method of claim 3 , wherein determining a pupil contour border, comprises:

determining contours in the first binary image; and

selecting a longest contour of the determined contours in the first binary image as a pupil contour border.

5 . The computer-implemented method of claim 4 , wherein the color value of the first binary image pixel of the first binary image is the fourth color value if a corresponding pixel in the semantically segmented eye image has a value greater than or equal to the fourth color value, and the third color value if the corresponding pixel in the semantically segmented eye image has a value not greater than or equal to the fourth color value.

6 . The computer-implemented method of claim 4 , comprising:

determining a pupil contour points bounding box enclosing the pupil contour border;

computing a pupil points area size as a diagonal of the pupil contours points bounding box; and

determining a pupil contour threshold based on the pupil points area size.

7 . The computer-implemented method of claim 6 , wherein the pupil contour threshold is a fraction multiplied by the pupil points area size, and wherein the fraction is in a range from 0.02 to 0.20.

8 . The computer-implemented method of claim 3 , comprising creating a third binary image comprising a plurality of pixels, wherein a color value of a third binary image pixel of the plurality of pixels of the third binary image is the third color value or the second color value.

9 . The computer-implemented method of claim 8 , wherein the color value of the third binary image pixel of the plurality of pixels of the third binary image is the third color value if a corresponding pixel in the semantically segmented eye image has a value greater than or equal to the third color value, and the second color value if the corresponding pixel in the semantically segmented eye image has a value not greater than or equal to the third color value.

10 . The computer-implemented method of claim 8 , wherein removing a plurality of pixels from the pupil contour border comprises, for a pupil contour border pixel of the pupil contour border:

determining a closest pixel in the third binary image that has a color value of the second color value and that is closest to the pupil contour border pixel;

determining a distance between the pupil contour border pixel and the closest pixel in the third binary image; and

removing the pupil contour border pixel from the pupil contour border if the distance between the pupil contour border pixel and the closest pixel in the third binary image is smaller than a pupil contour threshold.

11 . The computer-implemented method of claim 1 , wherein determining the iris contour using the semantically segmented eye image, comprises:

determining an iris contour border;

removing a plurality of pixels from the iris contour border; and

determining the iris contour as an ellipse from remaining pixels of the iris contour border.

12 . The computer-implemented method of claim 11 , wherein determining the iris contour border, comprises:

determining contours in the second binary image; and

selecting a longest contour of the determined contours in the second binary image as an iris contour border.

13 . The computer-implemented method of claim 12 , comprising:

determining an iris contour points bounding box enclosing the iris contour border;

computing an iris points area size as a diagonal of the iris contours points bounding box; and

determining an iris contour threshold based on the iris points area size.

14 . The computer-implemented method of claim 13 , wherein the iris contour threshold is a fraction multiple by the iris points area size, and wherein the fraction is in a range from 0.02 to 0.20.

15 . The computer-implemented method of claim 11 , wherein a color value of the second binary image pixel of the plurality of pixels of the second binary image is the third color value if a corresponding pixel in the semantically segmented eye image has a value greater than or equal to the third color value, and the second color value if the corresponding pixel in the semantically segmented eye image has a value not greater than or equal to the third color value.

16 . The computer-implemented method of claim 11 comprising creating a fourth binary image comprising a plurality of pixels, wherein a color value of a fourth binary image pixel of the plurality of pixels of the fourth binary image is the second color value or a first color value.

17 . The computer-implemented method of claim 16 , wherein a color value of a fourth binary image pixel of the plurality of pixels of the fourth binary image is the second color value if a corresponding pixel in the semantically segmented eye image has a value greater than or equal to the second color value, and the first color value if the corresponding pixel in the semantically segmented eye image has a value not greater than or equal to the second color value.

18 . The computer-implemented method of claim 16 , wherein removing a plurality of pixels from the iris contour border comprises, for an iris contour border pixel of the iris contour border:

determining a closest pixel in the fourth binary image that has a color value of the first color value and that is closest to the iris contour border pixel;

determining a distance between the iris contour border pixel and the closest pixel in the fourth binary image; and

removing the iris contour border pixel from the iris contour border if the distance between the iris contour border pixel and the closest pixel in the fourth binary image is smaller than an iris contour threshold.

19 . The computer-implemented method of claim 1 comprising determining a binary mask to cover an irrelevant area in the semantically segmented eye image.

20 . The computer-implemented method of claim 19 wherein determining the binary mask to cover the irrelevant area in the eye image, comprises:

creating a binary mask image comprising a plurality of pixels, wherein a binary mask image pixel of the binary mask image has a color value;

setting the color value of the binary mask image pixel to the third color value if a corresponding pixel in the semantically segmented eye image has a value greater than or equal to the third color value; and

setting the color value of the binary mask image pixel to the second color value if a corresponding pixel in the semantically segmented eye image has a value not greater than or equal to the third color value.

21 . The computer-implemented method of claim 19 wherein the irrelevant area comprises a portion of the semantically segmented eye image outside of an area defined by the iris contour.

22 . The computer-implemented method of claim 19 , comprising:

applying the binary mask to the semantically segmented eye image to generate a relevant eye image comprising a portion of the eye image that excludes the irrelevant area; and

calculating a biometric signature from the relevant eye image.

23 . The computer-implemented method of claim 22 , wherein the biometric signature comprises an iris code.

24 . The computer-implemented method of claim 19 , comprising creating a polar image of an iris of an eye in the eye image from the eye image using the pupil contour, the iris contour, and the binary mask to cover an irrelevant area in the semantically segmented eye image.

25 . The computer-implemented method of claim 1 , wherein receiving the semantically segmented eye image of an eye image comprising a plurality of pixels, comprises:

receiving an eye image;

processing the eye image using a convolution neural network to generate the semantically segmented eye image; and

processing the eye image using the convolution neural network to generate a quality estimation of the eye image,

wherein the convolution neural network comprises a segmentation tower and a quality estimation tower, wherein the segmentation tower comprises segmentation layers and shared layers,

wherein the quality estimation tower comprises quality estimation layers and the shared layers,

wherein a first output layer of the shared layers is connected to a first input layer of the segmentation tower and a second input layer of the segmentation tower,

wherein the first output layer of the shared layers is connected to an input layer of the quality estimation layer, and

wherein receiving the eye image comprises receiving the eye image by an input layer of the shared layers.

26 . The computer-implemented method of claim 1 , wherein:

a first color value corresponds to a background of the eye image;

the second color value corresponds to a sclera of an eye in the eye image;

the third color value corresponds to an iris of the eye in the eye image; and

the fourth color value corresponds to a pupil of the eye in the eye image.

Assignments (3)
SECURITY INTEREST Recorded Oct 20, 2025
From: MAGIC LEAP, INC.; MENTOR ACQUISITION ONE, LLC; MOLECULAR IMPRINTS, INC.
To: CITIBANK, N.A., AS COLLATERAL AGENT
Reel/Frame 073008/0696 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 30, 2023
From: SPIZHEVOY, ALEXEY; BADRINARAYANAN, VIJAY
To: MAGIC LEAP, INC.
Reel/Frame 064750/0704 →
EMPLOYMENT AGREEMENT Recorded Aug 30, 2023
From: KAEHLER, ADRIAN
To: MAGIC LEAP, INC.
Reel/Frame 064761/0340 →
Priority Claims (1)
RU RU2016138608 · Sep 29, 2016 · national
Continuity (4)
Continuation 17407763 · Aug 20, 2021
Continuation 16570418 · Sep 13, 2019
Continuation 15605567 · May 25, 2017
Related Publication 20240029269A1 · Jan 25, 2024
References Cited (206)
US 5291560A · Daugman · 1994 [cited by applicant]
US 5583795A · Smyth · 1996 [cited by applicant]
US 6850221B1 · Tickle · 2005 [cited by applicant]
US 7771049B2 · Knaan et al. · 2010 [cited by applicant]
US 7970179B2 · Tosa · 2011 [cited by applicant]
US 8098891B2 · Lv et al. · 2012 [cited by applicant]
US 8341100B2 · Miller et al. · 2012 [cited by applicant]
US 8345984B2 · Ji et al. · 2013 [cited by applicant]
US 8363783B2 · Gertner et al. · 2013 [cited by applicant]
US 8845625B2 · Angeley et al. · 2014 [cited by applicant]
US 8950867B2 · Macnamara · 2015 [cited by applicant]
US 9141916B1 · Corrado et al. · 2015 [cited by applicant]
US 9215293B2 · Miller · 2015 [cited by applicant]
US 9262680B2 · Nakazawa et al. · 2016 [cited by applicant]
US 9277207B2 · Ushiki et al. · 2016 [cited by applicant]
US 9310559B2 · Macnamara · 2016 [cited by applicant]
US 9348143B2 · Gao et al. · 2016 [cited by applicant]
US D758367S · Natsume · 2016 [cited by applicant]
US 9417452B2 · Schowengerdt et al. · 2016 [cited by applicant]
US 9430829B2 · Madabhushi et al. · 2016 [cited by applicant]
US 9791700B2 · Schowengerdt et al. · 2017 [cited by applicant]
US 10445881B2 · Spizhevoy et al. · 2019 [cited by applicant]
US 11100644B2 · Spizhevoy et al. · 2021 [cited by applicant]
US 20040130680A1 · Zhou et al. · 2004 [cited by applicant]
US 20060028436A1 · Armstrong · 2006 [cited by applicant]
US 20060088193A1 · Muller et al. · 2006 [cited by applicant]
US 20060147094A1 · Yoo · 2006 [cited by applicant]
US 20070081123A1 · Lewis · 2007 [cited by applicant]
US 20070140531A1 · Hamza · 2007 [cited by applicant]
US 20070274570A1 · Hamza · 2007 [cited by applicant]
US 20110182469A1 · Ji et al. · 2011 [cited by applicant]
US 20120127062A1 · Bar-Zeev et al. · 2012 [cited by applicant]
US 20120162549A1 · Gao et al. · 2012 [cited by applicant]
US 20120163678A1 · Du · 2012 [cited by examiner]
US 20130082922A1 · Miller · 2013 [cited by applicant]
US 20130117377A1 · Miller · 2013 [cited by applicant]
US 20130125027A1 · Abovitz · 2013 [cited by applicant]
US 20130208234A1 · Lewis · 2013 [cited by applicant]
US 20130242262A1 · Lewis · 2013 [cited by applicant]
US 20140071539A1 · Gao · 2014 [cited by applicant]
US 20140161325A1 · Bergen · 2014 [cited by examiner]
US 20140177023A1 · Gao et al. · 2014 [cited by applicant]
US 20140218468A1 · Gao et al. · 2014 [cited by applicant]
US 20140267420A1 · Schowengerdt · 2014 [cited by applicant]
US 20140270405A1 · Derakhshani et al. · 2014 [cited by applicant]
US 20140279774A1 · Wang et al. · 2014 [cited by applicant]
US 20140306866A1 · Miller et al. · 2014 [cited by applicant]
US 20140380249A1 · Fleizach · 2014 [cited by applicant]
US 20150016777A1 · Abovitz et al. · 2015 [cited by applicant]
US 20150103306A1 · Kaji et al. · 2015 [cited by applicant]
US 20150117760A1 · Wang et al. · 2015 [cited by applicant]
US 20150125049A1 · Taigman et al. · 2015 [cited by applicant]
US 20150134583A1 · Tamatsu et al. · 2015 [cited by applicant]
US 20150170002A1 · Szegedy et al. · 2015 [cited by applicant]
US 20150178939A1 · Bradski et al. · 2015 [cited by applicant]
US 20150205126A1 · Schowengerdt · 2015 [cited by applicant]
US 20150222883A1 · Welch · 2015 [cited by applicant]
US 20150222884A1 · Cheng · 2015 [cited by applicant]
US 20150268415A1 · Schowengerdt et al. · 2015 [cited by applicant]
US 20150278642A1 · Chertok et al. · 2015 [cited by applicant]
US 20150302652A1 · Miller et al. · 2015 [cited by applicant]
US 20150309263A2 · Abovitz et al. · 2015 [cited by applicant]
US 20150324974A1 · Garber · 2015 [cited by examiner]
US 20150326570A1 · Publicover et al. · 2015 [cited by applicant]
US 20150338915A1 · Publicover et al. · 2015 [cited by applicant]
US 20150346490A1 · TeKolste et al. · 2015 [cited by applicant]
US 20150346495A1 · Welch et al. · 2015 [cited by applicant]
US 20160011419A1 · Gao · 2016 [cited by applicant]
US 20160026253A1 · Bradski et al. · 2016 [cited by applicant]
US 20160034811A1 · Paulik et al. · 2016 [cited by applicant]
US 20160035078A1 · Lin et al. · 2016 [cited by applicant]
US 20160098844A1 · Shaji et al. · 2016 [cited by applicant]
US 20160104053A1 · Yin et al. · 2016 [cited by applicant]
US 20160104056A1 · He et al. · 2016 [cited by applicant]
US 20160135675A1 · Du et al. · 2016 [cited by applicant]
US 20160162782A1 · Park · 2016 [cited by applicant]
US 20160171300A1 · Takemoto · 2016 [cited by examiner]
US 20170053165A1 · Kaehler · 2017 [cited by applicant]
US 20180018451A1 · Spizhevoy et al. · 2018 [cited by applicant]
US 20180018515A1 · Spizhevoy et al. · 2018 [cited by applicant]
US 20180089834A1 · Spizhevoy et al. · 2018 [cited by applicant]
US 20200005462A1 · Spizhevoy et al. · 2020 [cited by applicant]
CN 103632137 · 2014 [cited by applicant]
CN 104408717 · 2015 [cited by applicant]
EP 2416279 · 2012 [cited by applicant]
JP 2004206444 · 2004 [cited by applicant]
JP 2009529200 · 2009 [cited by applicant]
JP 2015536519 · 2015 [cited by applicant]
WO WO2014182769 · 2014 [cited by applicant]
WO WO2015164807 · 2015 [cited by applicant]
WO WO2018063451 · 2018 [cited by applicant]
“Feature Extraction Using Convolution”, UFLDL, printed Sep. 1, 2016, in 3 pages. URL:http://deeplearning.stanford.edu/wiki/index.php/Feature_extraction_using_convolution. [cited by applicant]
“Machine Learning”, Wikipedia, printed Oct. 3, 2017, in 14 pages. URL: https://en.wikipedia.org/wiki/Machine_learning. [cited by applicant]
“Transfer Function Layers”, GitHub, Dec. 1, 2015, in 17 pages; accessed URL: http://github.com/torch/nn/blob/master/doc/transfer.md. [cited by applicant]
Adegoke et al., “Iris Segmentation: A Survey”, Int J Mod Engineer Res. (IJMER) (Jul./Aug. 2013) 3(4): 1885-1889. [cited by applicant]
Anthony, S., “MIT releases open-source software that reveals invisible motion and detail in video”, Extreme Tech, Feb. 28, 2013, as accessed Aug. 4, 2017, in 5 pages. [cited by applicant]
Arevalo J. et al., “Convolutional neural networks for mammography mass lesion classification”, in Engineering in Medicine and Biology Society (EMBC); 37th Annual International Conference IEEE, Aug. 25-29, 2015, pp. 797-… [cited by applicant]
ARToolKit: https://web.archive.org/web/20051013062315/http://www.hitl.washington.edu:80/artoolkit/documentation/hardware.htm, archived Oct. 13, 2005, 3 pages. [cited by applicant]
Aubry M. et al., “Seeing 3D chairs: exemplar part-based 2D-3D alignment using a large dataset of CAD models”, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (Jun. 23-28, 2014); Computer Vi… [cited by applicant]
Azuma, “A Survey of Augmented Reality,” Teleoperators and Virtual Environments 6, 4 (Aug. 1997), pp. 355-385. https://web.archive.org/web/20010604100006/http://www.cs.unc.edu/˜azuma/ARpresence.pdf. [cited by applicant]
Azuma, “Predictive Tracking for Augmented Realty,” TR95-007, Department of Computer Science, UNC—Chapel Hill, NC, Feb. 1995, 262 pages. [cited by applicant]
Badrinarayanan et al., “SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation”, IEEE (Dec. 8, 2015) arXiv:1511.00561v2 in 14 pages. [cited by applicant]
Bansal A. et al., “Marr Revisited: 2D-3D Alignment via Surface Normal Prediction”, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (Jun. 27-30, 2016) pp. 5965-5974. [cited by applicant]
Belagiannis V. et al., “Recurrent Human Pose Estimation”, In Automatic Face & Gesture Recognition; 12th IEEE International Conference—May 2017, ar Xiv:1605.02914v3; (Aug. 5, 2017) Open Access Version in 8 pages. [cited by applicant]
Bells et al., “Inside-Outside Net: Detecting Objects in Context with Skip Pooling and Recurrent Neural Networks”, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Jun. 27-30, 2016; pp. 2… [cited by applicant]
Biederman I., “Recognition-by-Components: A Theory of Human Image Understanding”, Psychol Rev. (Apr. 1987) 94(2): 115-147. [cited by applicant]
Bimber, et al., “Spatial Augmented Reality—Merging Real and Virtual Worlds,” 2005 https://web.media.mit.edu/˜raskar/book/BimberRaskarAugmentedRealityBook.pdf, 393 pages. [cited by applicant]
Bouget, J., “Camera Calibration Toolbox for Matlab” Cal-Tech, Dec. 2, 2013, in 5 pages. URL: https://www.vision.caltech.edu/bouguetj/calib_doc/index.html#parameters. [cited by applicant]
Bulat A. et al., “Human pose estimation via Convolutional Part Heatmap Regression”, arXiv e-print arXiv:1609.01743v1, Sep. 6, 2016 in 16 pages. [cited by applicant]
Camera calibration with OpenCV, OpenCV, retrieved May 5, 2016, in 7 pages. URL: http://docs.opencv.org/3.1.0/d4/d94/tutorial_camera_calibration.html#gsc.tab=0. [cited by applicant]
Carreira J. et al., “Human Pose Estimation with Iterative Error Feedback”, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Jun. 27-30, 2016, pp. 4733-4742. [cited by applicant]
Chatfield et al., “Return of the Devil in the Details: Delving Deep into Convolutional Nets”, arXiv eprint arXiv:1405.3531v4, Nov. 5, 2014 in 12 pages. [cited by applicant]
Chen X. et al., “3D Object Proposals for Accurate Object Class Detection”, in Advances in Neural Information Processing Systems, (2015) Retrieved from <http://papers.nips.cc/paper/5644-3dobjectproposals-for-accurate-obj… [cited by applicant]
Choy et al., “3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction”, arXiv; e-print arXiv:1604.00449v1, Apr. 2, 2016 in 17 pages. [cited by applicant]
Collet et al., “The MOPED framework: Object Recognition and Pose Estimation for Manipulation”, The International Journal of Robotics Research. (Sep. 2011) 30(10):1284-306; preprint Apr. 11, 2011 in 22 pages. [cited by applicant]
Crivellaro A. et al., “A Novel Representation of Parts for Accurate 3D Object Detection and 120 Tracking in Monocular Images”, In Proceedings of the IEEE International Conference on Computer Vision; Dec. 7-13, 2015 (pp.… [cited by applicant]
Dai J. et al., “Instance-aware Semantic Segmentation via Multi-task Network Cascades”, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; Jun. 27-30, 2016 (pp. 3150-3158). [cited by applicant]
Dai J. et al., “R-FCN: Object Detection via Region-based Fully Convolutional Networks”, in Advances in neural information processing systems; (Jun. 21, 2016) Retrieved from <https://arxiv.org/pdf/1605.06409.pdf in 11 pa… [cited by applicant]
Daugman, J. et al., “Epigenetic randomness, compleity and singularity of human iris patterns”, Proceedings of Royal Society: Biological Sciences, vol. 268, Aug. 22, 2001, in 4 pages. [cited by applicant]
Daugman, J., “How Iris Recognition Works”, IEEE Transactions on Circuits and Systems for Video Technology, vol. 14, No. 1, Jan. 2004, in 10 pages. [cited by applicant]
Daugman, J., “New Methods in Iris Recognition,” IEEE Transactions on Systems, Man, and Cybernetics—Part B: Cybernetics, vol. 37, No. 5, Oct. 2007, in 9 pages. [cited by applicant]
Daugman, J., “Probing the Uniqueness and Randomness of IrisCodes: Results From 200 Billion Iris Pair Comparisons, ” Proceedings of the IEEE, vol. 94, No. 11, Nov. 2006, in 9 pages. [cited by applicant]
Detone D. et al., “Deep Image Homography Estimation”, arXiv e-print arXiv:1606.03798v1, Jun. 13, 2016 in 6 pages. [cited by applicant]
Dwibedi et al., “Deep Cuboid Detection: Beyond 2D Bounding Boxes”, arXiv e-print arXiv:1611.1001 0v1; Nov. 30, 2016 in 11 pages. [cited by applicant]
Everingham M. et al., “The PASCAL Visual Object Classes (VOC) Challenge”, Int J Comput Vis (Jun. 2010) 88(2):303-38, 36 pages. [cited by applicant]
Farabet, C. et al., “Hardware Accelerated Convolutional Neural Networks for Synthetic Vision Systems”, Proceedings of the 2010 IEEE International Symposium (May 30-Jun. 2, 2010) Circuits and Systems (ISCAS), pp. 257-260. [cited by applicant]
Fidler S. et al., “3D Object Detection and Viewpoint Estimation with a Deformable 3D Cuboid Model”, in Proceedings of the 25th International Conference on Neural Information Processing Systems, (Dec. 3-6, 2012), pp. 611… [cited by applicant]
Fouhey D. et al., “Data-Driven 3D Primitives for Single Image Understanding”, Proceedings of the IEEE International Conference on Computer Vision, Dec. 1-8, 2013; pp. 3392-3399. [cited by applicant]
Geiger A. et al., “Joint 3D Estimation of Objects and Scene Layout”, In Advances in Neural Information Processing Systems 24; (Dec. 12-17, 2011) in 9 pages. [cited by applicant]
Gidaris S. et al., “Object detection via a multi-region & semantic segmentation-aware CNN model”, in Proceedings of the IEEE International Conference on Computer Vision; Dec. 7-13, 2015 (pp. 1134-1142). [cited by applicant]
Girshick R. et al., “Fast R-CNN”, Proceedings of the IEEE International Conference on Computer Vision; Dec. 7-13, 2015 (pp. 1440-1448). [cited by applicant]
Girshick R. et al., “Rich feature hierarchies for accurate object detection and semantic segmentation”, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Jun. 23-28, 2014 (pp. 580-587). [cited by applicant]
Gupta A. et al., “Blocks World Revisited: Image Understanding Using Qualitative Geometry and Mechanics”, in European Conference on Computer Vision; Sep. 5, 2010 in 14 pages. [cited by applicant]
Gupta A. et al., “From 3D Scene Geometry to Human Workspace”, in Computer Vision and Pattern Recognition (CVPR); IEEE Conference on Jun. 20-25, 2011 (pp. 1961-1968). [cited by applicant]
Gupta S. et al., “Aligning 3D Models to RGB-D Images of Cluttered Scenes”, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Jun. 7-12, 2015 (pp. 4731-4740). [cited by applicant]
Gupta S. et al., “Inferring 3D Object Pose in RGB-D Images”, arXiv e-print arXiv:1502.04652v1, Feb. 16, 2015 in 13 pages. [cited by applicant]
Gupta S. et al., “Learning Rich Features from RGB-D Images for Object Detection and Segmentation”, in European Conference on Computer Vision; (Jul. 22, 2014); Retrieved from <https://arxiv.org/pdf/1407.5736.pdf> in 16 p… [cited by applicant]
Han et al., “Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding”, arXiv e-print arX iv:1510.00149v5, Feb. 15, 2016 in 14 pages. [cited by applicant]
Hansen, D. et al., “In the Eye of the Beholder: A Survey of Models for Eyes and Gaze”, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 32, No. 3, Mar. 2010, in 23 pages. [cited by applicant]
Hartley R. et al., Multiple View Geometry in Computer Vision, 2nd Edition; Cambridge University Press, (Apr. 2004); in 673 pages. [cited by applicant]
He et al., “Delving Deep into Rectifiers: Surpassing Human-level Performance on ImageNet Classification”, arXiv: e-print arXiv:1502.01852v1, Feb. 6, 2015. [cited by applicant]
He et al., “Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition”, arXiv eprint arXiv:1406.4729v2; Aug. 29, 2014 in 14 pages. [cited by applicant]
Hedau V. et al., “Recovering Free Space of Indoor Scenes from a Single Image”, in Computer Vision and Pattern Recognition (CVPR), IEEE Conference Jun. 16-21, 2012 (pp. 2807-2814). [cited by applicant]
Hejrati et al., “Categorizing Cubes: Revisiting Pose Normalization”, Applications of Computer Vision (WACV), 2016 IEEE Winter Conference, Mar. 7-10, 2016 in 9 pages. [cited by applicant]
Hijazi, S. et al., “Using Convolutional Neural Networks for Image Recognition”, Tech Rep. (Sep. 2015) available on line URL: http://ip.cadence.com/uploads/901/cnn-wp-pdf, in 12 pages. [cited by applicant]
Hoffer et al., “Deep Metric Learning Using Triplet Network”, International Workshop on Similarity-Based Pattern Recognition [ICLR]; Nov. 25, 2015; [online] retrieved from the internet <https://arxv.org/abs/1412.6622>; p… [cited by applicant]
Hoiem D. et al., “Representations and Techniques for 3D Object Recognition and Scene Interpretation”, Synthesis Lectures on Artificial Intelligence and Machine Learning, Aug. 2011, vol. 5, No. 5, pp. 1-169; Abstract in … [cited by applicant]
Hsiao E. et al., “Making specific features less discriminative to improve point-based 3D object recognition”, in Computer Vision and Pattern Recognition (CVPR), IEEE Conference, Jun. 13-18, 2010 (pp. 2653-2660). [cited by applicant]
Huang et al., “Sign Language Recognition Using 3D Convolutional Neural Networks”, University of Science and Technology of China, 2015 IEEE International Conference on Multimedia and Expo. Jun. 29-Jul. 3, 2015, in 6 page… [cited by applicant]
Iandola F. et al., “SqueezeNet: Alex Net-level accuracy with 50x fewer parameters and <1 MB model size”, arXiv e-print arXiv:1602.07360v1, Feb. 24, 2016 in 13 pages. [cited by applicant]
International Preliminary Report on Patentability for PCT Application No. PCT/US2017/034482, issued Apr. 2, 2019, 7 pages. [cited by applicant]
International Search Report and Written Opinion for PCT Application No. PCT/US15/29679, mailed Jul. 6, 2017, 8 pages. [cited by applicant]
International Search Report and Written Opinion for PCT Application No. PCT/US17/29699, mailed Sep. 8, 2017, 20 pages. [cited by applicant]
International Search Report and Written Opinion for PCT Application No. PCT/US2017/034482, mailed Aug. 2, 2017, 10 pages. [cited by applicant]
International Search Report and Written Opinion for PCT Application No. PCT/US2017/054987, mailed Dec. 12, 2017, 17 pages. [cited by applicant]
International Search Report and Written Opinion for PCT Application No. PCT/US2017/061618, issued on Jan. 17, 2018. 18 pages. [cited by applicant]
International Search Report and Written Opinion for PCT Application No. PCT/US2017/48068, mailed Nov. 20, 2017, 14 pages. [cited by applicant]
Ioffe S. et al., “Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift”, International Conference on Machine Learning (Jun. 2015); pp. 448-456. [cited by applicant]
Jacob, “Eye Tracking in Advanced Interface Design,” Human-Computer Interaction Lab Naval Research Laboratory, Washington, D.C. /paper/in Virtual Environments and Advanced Interface Design, ed. by W. Barfield and T.A. Fu… [cited by applicant]
Jarrett et al., “What is the Best Multi-Stage Architecture for Object Recognition?,” In Computer Vision IEEE 12th International Conference Sep. 29-Oct. 2, 2009, pp. 2146-2153. [cited by applicant]
Ji, H. et al., “3D Convolutional Neural Networks for Human Action Recognition”, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 35:1, Jan. 2013, in 8 pages. [cited by applicant]
Jia et al., “3D-Based Reasoning with Blocks, Support, and Stability”, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; Jun. 23-28, 2013 in 8 pages. [cited by applicant]
Jia et al., “Caffe: Convolutional Architecture for Fast Feature Embedding”, arXiv e-print arXiv:1408.5093v1, Jun. 20, 2014 in 4 pages. [cited by applicant]
Jiang H. et al., “A Linear Approach to Matching Cuboids in RGBD Images”, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Jun. 23-28, 2013 (pp. 2171-2178). [cited by applicant]
Jillela et al., “An Evaluation of Iris Segmentation Algorithms in Challenging Periocular Images”, Handbook of Iris Recognition, Springer Verlag, Heidelberg (Jan. 12, 2013) in 28 pages. [cited by applicant]
Jou, et al., “Deep Cross Residual Learning for Multitask Visual Recognition,” arxiv.org. Cornell University Library, 201 OLIN Library XP080693514, Apr. 5, 2016, 9 pages. [cited by applicant]
Kar et al., “Category-specific object reconstruction from a single image”, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Jun. 7-12, 2015 (pp. 1966-1974). [cited by applicant]
Krizhevsky et al., “ImageNet Classification with Deep Convolutional Neural Networks”, Advances in Neural Information Processing Systems. Apr. 25, 2013, pp. 1097-1105. [cited by applicant]
Lavin, A. et al.: “Fast Algorithms for Convolutional Neural Networks”, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (Nov. 2016) arXiv:1509.09308v2, Nov. 10, 2015 in 9 pages. [cited by applicant]
Lee D. et al., “Geometric Reasoning for Single Image Structure Recovery”, in IEEE Conference Proceedings in Computer Vision and Pattern Recognition (CVPR) Jun. 20-25, 2009, pp. 2136-2143. [cited by applicant]
Lim J. et al., “FPM: Fine pose Parts-based Model with 3D Cad models”, European Conference on Computer Vision; Springer Publishing, Sep. 6, 2014, pp. 478-493. [cited by applicant]
Liu et al., “ParseNet: Looking Wider to See Better”, arXiv e-print arXiv:1506.04579v1; Jun. 15, 2015 in 9 pages. [cited by applicant]
Liu W. et al., “SSD: Single Shot MultiBo Detector”, arXiv e-print arXiv:1512.02325v5, Dec. 29, 2016 in 17 pages. [cited by applicant]
Long et al., “Fully Convolutional Networks for Semantic Segmentation”, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (Jun. 7-12, 2015) in 10 pages. [cited by applicant]
Mottalli, et al., “Flexible Image Segmentation and Quality Assessment For Real-Time Iris Recognition,” Image Processing (ICIP), 2009 16th IEEE International Conference, Nov. 7, 2009, 4 pages. [cited by applicant]
Office Action in Australian Appln. No. 2017337317, dated Dec. 15, 2021, 3 pages. [cited by applicant]
Open CV: “Camera calibration with OpenCV”, OpenCV, retrieved May 5, 2016, in 12 pages. URL: http://docs.opencv.org/2.4/doc/tutorials/calib3d/camera_calibration/camera_calibration.html. [cited by applicant]
OpenCV: “Camera Calibration and 3D Reconstruction”, OpenCV, retrieved May 5, 2016, in 51 pages. URL: http://docs.opencv.org/2.4/modules/calib3d/doc/camera_calibration_and_3d_reconstruction.html. [cited by applicant]
Pavlakos G. et al., “6-dof object pose from semantic keypoints”, in arXiv preprint Mar. 14, 2017; Retrieved from <http://www.cis.upenn.edu/˜kostas/mypub.dir/pavlakos17icra.pdf> in 9 pages. [cited by applicant]
Rastegari et al., “XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks”, arXiv e-print arXiv:1603.05279v4; Aug. 2, 2016 in 17 pages. [cited by applicant]
Redmon et al., “You Only Look Once: Unified, Real-Time Object Detection”, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (Jun. 27-30, 2016) pp. 779-788. [cited by applicant]
Ren et al., “Faster R-CNN: Towards real-time object detection with region proposal networks”, arxiv eprint arXiv:1506.01497v3; Jan. 6, 2016 in 14 pages. [cited by applicant]
Ren et al.: “On Vectorization of Deep Convolutional Neural Networks for Vision Tasks,” AAAI, arXiv: eprint arXiv:1501.07338v1, Jan. 29, 2015 in 8 pages. [cited by applicant]
Roberts L. et al., “Machine Perception of Three-Dimensional Solids”, Doctoral Thesis MIT; Jun. 1963 in 82 pages. [cited by applicant]
Rubinstein, M., “Eulerian Video Magnification”, YouTube, published May 23, 2012, as archived Sep. 6, 2017, in 13 pages (with video transcription). URL: https://web.archive.org/web/20170906180503/https://www.youtube.com/… [cited by applicant]
Savarese et al., “3D generic object categorization, localization and pose estimation”, in Computer Vision, IEEE 11th International Conference; Oct. 14-21, 2007, in 8 pages. [cited by applicant]
Saxena A., “Convolutional Neural Networks (CNNS): An Illustrated E planation”, Jun. 29, 2016 in 16 pages; Retrieved from <http://xrds.acm.org/blog/2016/06/convolutional-neural-networks-cnns-illustrated-explanation/>. [cited by applicant]
Schroff et al., “FaceNet: A unified embedding for Face Recognition and Clustering”, arXiv e-print arXiv:1503.03832v3, Jun. 17, 2015 in 10 pages. [cited by applicant]
Shafiee et al., “ISAAC: A Convolutional Neural Network Accelerator with In-Situ Analog Arithmetic in Crossbars”, ACM Sigarch Comp. Architect News (Jun. 2016) 44(3):14-26. [cited by applicant]
Shao et al., “Imagining the Unseen: Stability-based Cuboid Arrangements for Scene Understanding”, ACM Transactions on Graphics. (Nov. 2014) 33(6) in 11 pages. [cited by applicant]
Simonyan et al., “Very deep convolutional networks for large-scale image recognition”, arXiv e-print arXiv:1409.1556v6, Apr. 10, 2015 in 14 pages. [cited by applicant]
Song et al., “Deep Sliding Shapes for Amodal 3D Object Detection in RGB-D Images”, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Jun. 27-30, 2016 (pp. 808-816). [cited by applicant]
Song et al., “Sliding Shapes for 3D Object Detection in Depth Images”, in European Conference on Computer Vision, (Sep. 6, 2014) Springer Publishing (pp. 634-651 ). [cited by applicant]
Su et al., “Render for CNN: Viewpoint Estimation in Images Using CNNs Trained with Rendered 3D Model Views”, in Proceedings of the IEEE International Conference on Computer Vision, Dec. 7-13, 2015 (pp. 2686-2694). [cited by applicant]
Szegedy et al., “Going deeper with convolutions”, arXiv:1409.4842v1, Sep. 17, 2014 in 12 pages. [cited by applicant]
Szegedy et al., “Rethinking the Inception Architecture for Computer Vision”, arXiv e-print arXIV:1512.00567v3, Dec. 12, 2015 in 10 pages. [cited by applicant]
Tanriverdi and Jacob, “Interacting With Eye Movements in Virtual Environments,” Department of Electrical Engineering and Computer Science, Tufts University, Medford, MA—paper/Proc. ACM CHI 2000 Human Factors in Computin… [cited by applicant]
Tulsiani S. et al., “Viewpoints and Keypoints”, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; Jun. 7-12, 2015 (pp. 1510-1519). [cited by applicant]
Villanueva, A. et al., “A Novel Gaze Estimation System with One Calibration Point”, IEEE Transactions on Systems, Man, and Cybernetics-Part B:Cybernetics, vol. 38:4, Aug. 2008, in 16 pages. [cited by applicant]
Wikipedia: “Convolution”, Wikipedia, accessed Oct. 1, 2017, in 17 pages. URL: https://en.wikipedia.org/wiki/Convolution. [cited by applicant]
Wikipedia: “Deep Learning”, Wikipedia, printed Oct. 3, 2017, in 23 pages. URL: https://en.wikipedia.org/wiki/Deep_learning. [cited by applicant]
Wilczkowiak et al., “Using Geometric Constraints Through Parallelepipeds for Calibration and 3D Modelling”, IEEE Transactions on Pattern Analysis and Machine Intelligence—No. 5055 (Nov. 2003) 27(2) in 53 pages. [cited by applicant]
Wu et al., “Single Image 3D Interpreter Network”, arXiv e-print arXiv:1604.08685v2, Oct. 4, 2016 in 18 pages. [cited by applicant]
Xiao et al., “Localizing 3D cuboids in single-view images”, in Advances in Neural Information Processing Systems 25. F. Pereira et al. [Eds.] Apr. 2013 in 9 pages. [cited by applicant]
Yang et al., “Articulated human detection with flexible mixtures of parts”, IEEE Transactions on Pattern Analysis and Machine Intelligence. Dec. 2013; 35(12):2878-90. [cited by applicant]
Zheng et al., “Interactive Images: Cuboid Proxies for Smart Image Manipulation”, ACM Trans Graph. (Jul. 2012) 31(4):99-109. [cited by applicant]
Cited By (1)
US 12,608,837