Granted Patent
B2
US 12,288,403 · App. 18/611,462 · Granted Apr 29, 2025
Occupant attentiveness and cognitive load monitoring for autonomous and semi-autonomous driving applications
Inventors:
Nuri Murat Arar (Zurich, CH); Niranjan Avadhanam (Saratoga, CA); Yuzhuo Ren (San Jose, CA)
Assignee:
NVIDIA Corporation
G06V20/597B60W30/0956B60W40/08B60W50/14B60W60/001G06V20/56B60W30/09B60W2540/22B60W2540/221B60W2540/223B60W2540/225
View Patent ↗
Loading inventors, assignments & file history…
Abstract
In various examples, estimated field of view or gaze information of a user may be projected external to a vehicle and compared to vehicle perception information corresponding to an environment outside of the vehicle. As a result, interior monitoring of a driver or occupant of the vehicle may be used to determine whether the driver or occupant has processed or seen certain object types, environmental conditions, or other information exterior to the vehicle. For a more holistic understanding of the state of the user, attentiveness and/or cognitive load of the user may be monitored to determine whether one or more actions should be taken. As a result, notifications, AEB system activations, and/or other actions may be determined based on a more complete state of the user as determined based on cognitive load, attentiveness, and/or a comparison between external perception of the vehicle and estimated perception of the user.
Claims (78)
1. A method comprising:
generating, in a coordinate system and based at least on first sensor data obtained using one or more first sensors of a vehicle, a representation of at least a portion of a field of view of a user;
determining, in the coordinate system and based at least on second sensor data obtained using one or more second sensors of the vehicle, an object location of at least one object;
projecting the representation towards the object location; and
generating a notification based at least on the projecting.
2. The method of claim 1 , wherein the generating the representation comprises:
determining, based at least on the first sensor data, a gaze direction of the user; and
determining, in the coordinate system, the representation of the at least the portion of the field of view based at least on the gaze direction of the user.
3. The method of claim 1 , further comprising:
determining that an amount of overlap between the representation and the object location is less than a threshold amount,
wherein the generating the notification is further based at least on the amount of overlap being less than the threshold amount.
4. The method of claim 1 ,
wherein the representation is projected external to the vehicle and towards the object location.
5. The method of claim 1 , further comprising:
determining that a cognitive load value associated with the user is greater than a threshold value,
wherein the generating the notification is further based at least on the cognitive load value being greater than the threshold value.
6. The method of claim 5 , further comprising:
monitoring one or more eye characteristics of the user of the vehicle, at least one eye characteristic of the one or more eye characteristics including one or more eye movements or one or more eye measurements; and
determining the cognitive load value of the user based at least on the one or more eye characteristics of the user.
7. The method of claim 1 , further comprising:
determining that an attentiveness value associated with the user is less than a threshold value,
wherein the generating the notification is further based at least on the attentiveness value being less than the threshold value.
8. The method of claim 7 , further comprising:
monitoring eye movement of the user of the vehicle to determine one or more eye behaviors of the user, at least one eye behavior of the one or more eye behaviors including one or more gaze patterns, one or more saccade velocities, one or more fixation behaviors, or one or more smooth pursuit behaviors; and
determining the attentiveness value of the user based at least on the one or more eye behaviors of the user.
9. The method of claim 1 , wherein:
the one or more first sensors include a first field of view associated with an interior compartment of the vehicle; and
the one or more second sensors include a second field of view associated with an exterior of the vehicle.
10. The method of claim 1 , wherein the coordinate system is a three-dimensional (3D) world space coordinate system.
11. The method of claim 1 , wherein the generating the representation of the at least the portion of the field of view of the user comprises:
determining, based at least on the first sensor data, one or more eye characteristics associated with the user; and
generating, in the coordinate system and based at least on the one or more eye characteristics, the representation of the at least the portion of the field of view of the user.
12. A system comprising:
one or more processors to:
determine, in a coordinate system and based at least on first sensor data obtained using one or more first sensors of a vehicle, a representation of at least a portion of a field of view of a user;
determine, in the coordinate system and based at least on second sensor data obtained using one or more second sensors of the vehicle, an object location of at least one object;
determine that an amount of overlap between the representation and the object location is less than a threshold amount; and
cause, based at least on the amount of overlap being less than the threshold amount, one or more operations associated with the vehicle to occur.
13. The system of claim 12 , wherein the determination of the representation comprises:
determining, based at least on the first sensor data, a gaze direction of the user; and
determining, in the coordinate system, the representation of the at least the portion of the field of view based at least on the gaze direction of the user.
14. The system of claim 12 , wherein:
the determination of the representation uses one or more first machine learning models that process the first sensor data; and
the determination of the object location uses one or more second machine learning models that process the second sensor data.
15. The system of claim 12 , wherein the one or more processors are further to:
project the representation external to the vehicle and towards the object location,
wherein the amount of overlap is determined based at least on the representation being projected towards the object location.
16. The system of claim 12 , wherein the one or more processors are further to:
determine that a cognitive load value associated with the user is greater than a threshold value,
wherein the one or more operations associated with the vehicle are further caused to occur based at least on the cognitive load value being greater than the threshold value.
17. The system of claim 12 , wherein the one or more processors are further to:
determine that an attentiveness value associated with the user is less than a threshold value,
wherein the one or more operations associated with the vehicle are further caused to occur based at least on the attentiveness value being less than the threshold value.
18. The system of claim 12 , wherein the system is comprised in at least one of:
a control system for an autonomous or semi-autonomous machine;
a perception system for an autonomous or semi-autonomous machine;
a system for performing simulation operations;
a system for performing deep learning operations;
a system implemented using an edge device;
a system incorporating one or more virtual machines (VMs);
a system implemented using a robot;
a system implemented at least partially in a data center; or
a system implemented at least partially using cloud computing resources.
19. One or more processors comprising:
one or more processing units to:
determine at least one object location and a representation of a field of view of a user, wherein the representation is determined based at least on first sensor data obtained using one or more first sensors of a vehicle and the object location is determined based at least on second sensor data obtained using one or more second sensors of the vehicle;
determine an amount of overlap between the representation and the object location; and
cause one or more operations associated with the vehicle to occur based at least on the amount of overlap.
20. The one or more processors of claim 19 , wherein the one or more processors are comprised in at least one of:
a control system for an autonomous or semi-autonomous machine;
a perception system for an autonomous or semi-autonomous machine;
a system for performing simulation operations;
a system for performing deep learning operations;
a system implemented using an edge device;
a system incorporating one or more virtual machines (VMs);
a system implemented using a robot;
a system implemented at least partially in a data center; or
a system implemented at least partially using cloud computing resources.
Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST
Recorded Mar 22, 2024
From: ARAR, NURI MURAT; AVADHANAM, NIRANJAN; REN, YUZHUO
To: NVIDIA CORPORATION
Reel/Frame 066865/0786 →
Continuity (2)
References Cited (400)
US 2631498A
· Barkley
· 1953
[cited by applicant]
US 5909190A
· Lo et al.
· 1999
[cited by applicant]
US 7409295B2
· Paradie
· 2008
[cited by applicant]
US 7693683B2
· Ihara
· 2010
[cited by applicant]
US 7842869B2
· Adams et al.
· 2010
[cited by applicant]
US 8204542B2
· Liao et al.
· 2012
[cited by applicant]
US 8204642B2
· Tanaka et al.
· 2012
[cited by applicant]
US 8385676B2
· Weinberger et al.
· 2013
[cited by applicant]
US 8503801B2
· Schiller et al.
· 2013
[cited by applicant]
US 8554715B2
· Kraaij et al.
· 2013
[cited by applicant]
US 8837819B1
· Lees et al.
· 2014
[cited by applicant]
US 8929683B2
· Gallo et al.
· 2015
[cited by applicant]
US 9087297B1
· Filippova et al.
· 2015
[cited by applicant]
US 9098751B2
· Hilldore et al.
· 2015
[cited by applicant]
US 9373057B1
· Erhan et al.
· 2016
[cited by applicant]
US 9623905B2
· Shashua et al.
· 2017
[cited by applicant]
US 9701307B1
· Newman et al.
· 2017
[cited by applicant]
US 9710714B2
· Chen et al.
· 2017
[cited by applicant]
US 9721471B2
· Chen et al.
· 2017
[cited by applicant]
US 9738125B1
· Brickley et al.
· 2017
[cited by applicant]
US 9911050B2
· Lynam et al.
· 2018
[cited by applicant]
US 20090097704A1
· Savidge et al.
· 2009
[cited by applicant]
US 20170046178A1
· Singh
· 2017
[cited by examiner]
US 20180005057A1
· Lee et al.
· 2018
[cited by applicant]
US 20180136332A1
· Barfield, Jr. et al.
· 2018
[cited by applicant]
US 20180158244A1
· Ybanez Zepeda et al.
· 2018
[cited by applicant]
US 20180203959A1
· Refsnaes et al.
· 2018
[cited by applicant]
US 20180349746A1
· Vallespi-Gonzalez
· 2018
[cited by applicant]
US 20180373980A1
· Huval
· 2018
[cited by applicant]
US 20190061771A1
· Bier et al.
· 2019
[cited by applicant]
US 20190071101A1
· Emura et al.
· 2019
[cited by applicant]
US 20190120640A1
· Ho et al.
· 2019
[cited by applicant]
US 20190129831A1
· Goldberg
· 2019
[cited by applicant]
US 20190179979A1
· Melick
· 2019
[cited by applicant]
US 20190370580A1
· Aoi et al.
· 2019
[cited by applicant]
US 20200207358A1
· Katz et al.
· 2020
[cited by applicant]
US 20210300379A1
· Hackeloeer et al.
· 2021
[cited by applicant]
CN 102656613A
· 2012
[cited by applicant]
CN 102712317A
· 2012
[cited by applicant]
CN 105835874A
· 2016
[cited by applicant]
CN 107415938A
· 2017
[cited by applicant]
CN 108068821A
· 2018
[cited by applicant]
CN 110211586A
· 2019
[cited by applicant]
CN 110626356A
· 2019
[cited by applicant]
DE 102015221920A1
· 2017
[cited by applicant]
DE 102015226762A1
· 2017
[cited by applicant]
EP 1510973A2
· 2005
[cited by applicant]
EP 1930863A2
· 2008
[cited by applicant]
EP 1930868A1
· 2008
[cited by applicant]
EP 2384009A2
· 2011
[cited by applicant]
EP 3171297A1
· 2017
[cited by applicant]
EP 3441909A1
· 2019
[cited by applicant]
JP 2003308522A
· 2003
[cited by applicant]
JP 2012187178A
· 2012
[cited by applicant]
JP 2015194884A
· 2015
[cited by applicant]
JP 2019165391A
· 2019
[cited by applicant]
KR 1020120009590A
· 2012
[cited by applicant]
RU 2358319C2
· 2009
[cited by applicant]
WO 2012011713A2
· 2012
[cited by applicant]
WO 2016183074A1
· 2016
[cited by applicant]
WO 2017177128A1
· 2017
[cited by applicant]
WO 2017220705A1
· 2017
[cited by applicant]
WO 2018002910A1
· 2018
[cited by applicant]
WO 2018102717A1
· 2018
[cited by applicant]
Murat Arar, Nuri; First Office Action for Chinese Patent Application No. 202111221654.3, filed Oct. 20, 2021, mailed Dec. 13, 2023, 14 pgs.
[cited by applicant]
Roche, Jason; International Search Report and Written Opinion for PCT Application No. PCT/US2021/029411, filed Apr. 27, 2021, mailed Nov. 4, 2021, 16 pgs.
[cited by applicant]
Roche, Jason; Invitation to Pay Additional Fees for PCT Application No. PCT/US2021/029411, filed Apr. 27, 2021, mailed Sep. 13, 2021, 11 pgs.
[cited by applicant]
He, et al.; (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 770-778).
[cited by applicant]
Arar, Nuri Murat; Second Office Action for Chinese Patent Application No. 202111221654.3, filed Oct. 20, 2021, mailed Jun. 3, 2024, 12 pgs.
[cited by applicant]
Non-Final Office Action for U.S. Appl. No. 16/544,442, filed Aug. 19, 2019, mailed Jan. 29, 2021, 8 pgs.
[cited by applicant]
Otice of Allowance for U.S. Appl. No. 16/544,442, filed Aug. 19, 2019, mailed Jun. 11, 2021, 8 pgs.
[cited by applicant]
Jain, Anshul; Non-Final Office Action for U.S. Appl. No. 16/363,648, filed Mar. 25, 2019, mailed Feb. 24, 2021, 13 pgs.
[cited by applicant]
Sivaraman, Sakthivel; Notice of Allowance for U.S. Appl. No. 16/907,125, filed Jun. 19, 2020, mailed Aug. 19, 2021, 9 pgs.
[cited by applicant]
Jain, Anshul; Final Office Action for U.S. Appl. No. 16/363,648, filed Mar. 25, 2019, mailed Sep. 9, 2021, 11 pgs.
[cited by applicant]
Jain, Anshul; Non-Final Office Action for U.S. Appl. No. 16/363,648, filed Mar. 25, 2019, mailed Jan. 5, 2022, 15 pgs.
[cited by applicant]
Puri, Nishant; Non-Final Office Action for U.S. Appl. No. 17/010,205, filed Sep. 2, 2020, mailed Mar. 1, 2022, 14 pgs.
[cited by applicant]
Arar, Nuri Murat; Notice of Allowance for U.S. Appl. No. 17/004,252, filed Aug. 27, 2020, mailed Mar. 4, 2022, 9 pgs.
[cited by applicant]
Jain, Anshul; Final Office Action for U.S. Appl. No. 16/363,648, filed Mar. 25, 2019, mailed May 23, 2022, 17 pgs.
[cited by applicant]
Arar, Nuri Murat; Notice of Allowance for U.S. Appl. No. 17/004,252, filed Aug. 27, 2020, mailed Jun. 8, 2022, 9 pgs.
[cited by applicant]
Fridman, L., Langhans, P., Lee, J., & Reimer, B. (2015). Driver gaze estimation without using eye movement. arXiv preprint arXiv:1507.04760, 2(4).
[cited by applicant]
Naqvi, R. A., Arsalan, M., Batchuluun, G., Yoon, H. S., & Park, K. R. (2018). Deep learning-based gaze detection system for automobile drivers using a NIR camera sensor. Sensors, 18(2), 456.
[cited by applicant]
Vora, S., Rangesh, A., & Trivedi, M. M. (2018). Driver gaze zone estimation using convolutional neural networks: A general framework and ablative analysis. IEEE Transactions on Intelligent Vehicles, 3(3), 254-265.
[cited by applicant]
Chen, S., & Epps, J. (2013). Automatic classification of eye activity for cognitive load measurement with emotion interference. Computer methods and programs in biomedicine, 110(2), 111-124.
[cited by applicant]
Fridman, L., Reimer, B., Mehler, B., & Freeman, W. T. (Apr. 2018). Cognitive load estimation in the wild. In Proceedings of the 2018 chi conference on human factors in computing systems (pp. 1-9).
[cited by applicant]
U.S. Appl. No. 62/948,789, filed Dec. 16, 2019.
[cited by applicant]
U.S. Appl. No. 62/948,793, filed Dec. 16, 2019.
[cited by applicant]
Hassner, T., et al., “Effective Face Frontalization in Unconstrained Images”, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2015) 10 pgs.
[cited by applicant]
Kahou, S.E., et al.; “Combining Modality Specific Deep Neural Networks for Emotion Recognition in Video”; In Proceedings of the 15th ACM on International Conference on Multimodal Interaction (ICMI), Dec. 9-13, 2013; 8 p…
[cited by applicant]
Liu, Y., et al.; “Exploring Disentagled Feature Representation Beyond Face Identification”; In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, 10 pgs.
[cited by applicant]
Park, S., et al.; “Few-Shot Adaptive Gaze Estimation”; arXiv:1905.01941v2, Oct. 14, 2019, 13 pgs.
[cited by applicant]
Yang, T-Y., et al.; “FSA-Net: Learning Fine-Grained Structure Aggregation for Head Pose Estimation from a Single Image”; In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019, 10 pgs.
[cited by applicant]
Zhu, M., et al.; “Robust Facial Landmark Detection via Occlusion-adaptive Deep Networks”; In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019, 11 pgs.
[cited by applicant]
Zhu, X., et al.; “Face Alignment in Full Pose Range: A 3D Total Solution”; IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 41, No. 1, 2017; 14 pgs.
[cited by applicant]
Liang, Z., et al.; “Learning for Disparity Estimation through Feature Constancy”, Computer Vision and Pattern Recognition, pp. 2811-2820 (2018).
[cited by applicant]
Liu., et al., “Learning Depth from Single Monocular Images Using Deep Convolutional Neural Fields”; IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 1-16 (2015).
[cited by applicant]
Mayer., et al.; “A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation”; IEEE Conference on Computer Vision and Pattern Recognition, pp. 1-14 (2016).
[cited by applicant]
Mikic, I. et al.; “Human Body Model Acquisition and Tracking Using Voxel Data”, International Journal of Computer Vision, vol. 53, No. 3, pp. 199-223 (2003).
[cited by applicant]
Park, J J., et al.; “DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation”; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 165-174 (2019).
[cited by applicant]
Palmisano, S., et al.; “Stereoscopic perception of real depths at large distances”; Journal of Vision, vol. 10, No. 6, pp. 1-16 (Jun. 2010).
[cited by applicant]
Seki, A., et al., “Patch Based Confidence Prediction for Dense Disparity Map”; British Machine Vision Conference, pp. 1-13 (2016).
[cited by applicant]
Seki, A., et al., “SGM-Nets: Semi-Global Matching with Neural Networks”, IEEE Conference on computerVision and Pattern Recognition, pp. 231-240 (2017).
[cited by applicant]
Shaked., et al,; “Improved Stereo Matching With Constant Highway Networks and Reflective Confidence Learning”, IEEE Conference on Computer Vision and Pattern Recognition, pp. 1-13 (2016).
[cited by applicant]
Wang, Z., “Image quality assessment: from error visibility to structural similarity”, IEEE Transactions on Image Processing, vol. 13, No. 4, pp. 1-14, (Apr. 2014).
[cited by applicant]
Wu, et al.; “Automatic background filtering and lane identification with roadside LiDAR data”, 2017 IEEE 20th International Conference on Intelligent Transportation Systems (ISTC), pp. 1-6 (2017).
[cited by applicant]
Zbontar, J., et al., “Stereo Matching by Training a Convolutional Neural Network to Compare Image Patches”, Journal of Machine Learning Research, vol. 17, pp. 1-32 (2016).
[cited by applicant]
Zhao, et al., “Loss Functions for Image Restoration With Neural Networks”, IEEE Transactions on Computational Imaging, vol. 3, No. 1, pp. 1-11 (Mar. 2017).
[cited by applicant]
Invitation to pay additional fees received for PCT Application No. PCT/US2020/028116, mailed on Jul. 21, 2020, 12 pgs.
[cited by applicant]
International Search Report and Written Opinion in International Patent Application No. PCT/US2020/021894 mailed Aug. 3, 2020, 14 pgs.
[cited by applicant]
International Search Report and Written Opinion received for PCT Patent Application No. PCT/US2020/028116, mailed on Sep. 11, 2020, 17 pgs.
[cited by applicant]
International Preliminary Report on Patentability received for PCT Application No. PCT/US2020/021894, mailed Sep. 23, 2021, 11 pgs.
[cited by applicant]
International Preliminary Report on Patentability received for PCT Application No. PCT/US2020/028116, mailed on Nov. 4, 2021, 14 pgs.
[cited by applicant]
“Methods for High-Precision, High-Accuracy Lane Detection in Autonomous Driving Applications”, U.S. Appl. No. 62/699,669, filed Jul. 17, 2018.
[cited by applicant]
“Distance to Obstacle Detection in Autonomous Driving Application”, U.S. Appl. No. 62/786,188, filed Dec. 28, 2018.
[cited by applicant]
“Detection and Classification of Wait Conditions in Autonomous Driving Applications”, U.S. Appl. No. 62/816,838, filed Mar. 11, 2019.
[cited by applicant]
“Intersection Detection and Handling Using Live Perception in Autonomous Driving Application”, U.S. Appl. No. 62/839,155, filed Apr. 26, 2019.
[cited by applicant]
“Intersection Contention Area Detection Using Live Perception in Autonomous Driving Applications”, U.S. Appl. No. 62/866,158, filed Jun. 25, 2019.
[cited by applicant]
Allison, R.S., et al., “Binocular depth discrimination and estimation beyond interaction space”, Journal of Vision, vol. 9, No. 1, pp. 1-14 (Jan. 2009).
[cited by applicant]
Borland, D., and Taylor II, R.M., “Rainbow Color Map {Still} Considered Harmful”, IEEE Computer Graphics and Applications, vol. 27, No. 2, pp. 1-17 (Mar./Apr. 2007).
[cited by applicant]
Chen, J., et al., “FOAD: Fast Optimization-based Autonomous Driving Motion Planner”, 2018 Annual American control Conference (ACC), IEEE, pp. 1-8 (Jun. 27-29, 2018).
[cited by applicant]
Clevert, D.A., et al., “Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)”, ICLR 2016, pp. 1-14 (Feb. 22, 2016).
[cited by applicant]
Cormack, R.H., “Stereoscopic depth perception at far viewing distances”, Perception & Psychophysics, vol. 35, No. 5, pp. 423-428 (Sep. 1984).
[cited by applicant]
Eigen, D., et al., “Depth Map Prediction from a Single Image using a Multi-Scale Deep Network”, NIPS, pp. 1-9, (2014).
[cited by applicant]
Fukunaga, K., and Hostetler, L., “The estimation of the gradient of a density function, with applications in pattern recognition”, IEEE Transactions on Information Theory, vol. 21, No. 1, pp. 32-40 (Jan. 1975).
[cited by applicant]
Garg, R., et al., “Unsupervised CNN for Single View Depth Estimation: Geometry to the Rescue”, ECCV 2016, pp. 1-16 (Jul. 29, 2016).
[cited by applicant]
Geiger, A., et al., “Vision meets Robotics: The KITTI Dataset”, The International Journal of Robotics Research, pp. 1-6 (2013).
[cited by applicant]
Gidaris, S., and Komodakis, N., “Detect, Replace, Refine: Deep Structured Prediction for Pixel Wise Labeling”, computerVision and Pattern Recognition, pp. 1-21 (Dec. 14, 2016).
[cited by applicant]
Gregory, R.L., “Eye and brain: The psychology of seeing”, World University Library, p. 130 (1966) (Part 1).
[cited by applicant]
Gregory, R. L., “Eye and brain: The psychology of seeing”, World University Library, pp. 130 (1966) (Part 2).
[cited by applicant]
Guney, F., et al., Displets: Resolving Stereo Ambiguities using Object Knowledge:, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1-11 (Jun. 7-12, 2015).
[cited by applicant]
Hartley, R., and Zisserman, A., “Multiple View Geometry in Computer Vision”, Cambridge University Press, pp. 1-48 (2004).
[cited by applicant]
Hibbard, P. B., et al., “Magnitude, precision, and realism of depth perception in stereoscopic vision”, Cognitive Research: Principles and Implications, vol. 2, pp. 1-11 (2017).
[cited by applicant]
Jaderberg, M., et al., “Spatial Transformer Networks”, NIPS, pp. 1-9 (2015).
[cited by applicant]
Kovesi, P., “Good Colour Maps: How to Design Them” arXiv:1509.03700, pp. 1-42 (Sep. 12, 2015).
[cited by applicant]
Kutulakos, K. N., and Seitz, S. M., “A Theory of Shape by Space Carving” International Journal of Computer Vision, vol. 38, No. 3, pp. 199-218 (2000).
[cited by applicant]
Kuznietsov, Y., et al., “Semi-Supervised Deep Learning for Monocular Depth Map Prediction”, Computer Vision anc Pattern Recognition, pp. 6647-6655 (2017).
[cited by applicant]
Laurentini, A., “How far 3D shapes can be understood from 2D silhouettes”, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 17, No. 2, pp. 188-195 (Feb. 1995).
[cited by applicant]
Levin, C. A., and Haber, R. N., “Visual angle as a determinant of perceived interobject distance” Perception & Psychophysics vol. vol. 54, No. 2, pp. 250-259 (Mar. 1993).
[cited by applicant]
“What are deconvolutional layers?”, Data Science Stack Exchange, Retrieved from Internet URL: https://datascience.stackexchange.com/questions/6107/what%E2%80%90are%E2%80%90deconvolutional%E280%90layers, accessed on Feb.…
[cited by applicant]
Dipietro, R., “A Friendly Introduction to Cross-Entropy Loss,” Version 0.1, Retrieved from Internet URL: https://rdipietro.github.io/friendly-intro-to-cross-entropy-loss/, accessed Feb. 21, 2019, 10 pgs. (May 2, 2016).
[cited by applicant]
Foley, D., and Danskin, J., “Ultra-Performance Pascal GPU and NVLink Interconnect”, IEEE Computer Society, IEEE Micro, vol. 37, No. 2, pp. 1-11 (2017).
[cited by applicant]
Invitation to pay additional fees received for PCT Application No. PCT/US2019/018348, mailed on May 29, 2019, 18 pgs.
[cited by applicant]
Invitation to pay additional fees received for PCT Application No. PCT/US2019/019656, mailed on May 31, 2019, 9 pgs.
[cited by applicant]
Invitation to pay additional fees received for PCT Application No. PCT/US2019/022592, mailed on Jun. 26, 2019, 9 pgs.
[cited by applicant]
Du, L., and Du, Y., “Hardware Accelerator Design for Machine Learning,” Machine Learning—Advanced Techniques and Emerging Applications, Retrieved from Internet URL: https://www.intechopen.com/chapters/58659, pp. 1-14.
[cited by applicant]
Notice of Allowance dated Dec. 14, 2021 in U.S. Appl. No. 16/728,595, 8 pages.
[cited by applicant]
Yang, Yiilin; Notice of Allowance for U.S. Appl. No. 16/728,598, filed Jul. 16, 2021, mailed Jul. 16, 2021, 9 pgs.
[cited by applicant]
Kwon, Junghyun; Notice of Allowance for U.S. Appl. No. 16/813,306, filed Mar. 9, 2020, mailed Oct. 7, 2021, 9 pgs.
[cited by applicant]
Yang, Yilin; Non-Final Office Action for U.S. Appl. No. 16/728,595, filed Dec. 27, 2019, mailed May 13, 2021, 18 pgs.
[cited by applicant]
Kwon, Junghyun; Notice of Allowance for U.S. Appl. No. 16/813,306, filed Mar. 9, 2020, mailed Jul. 2, 2021, 13 pgs.
[cited by applicant]
International Search Report and Written Opinion received for PCT Patent Application No. PCT/US2019/068764, mailed on Apr. 22, 2020, 15 pgs.
[cited by applicant]
International Search Report and Written Opinion received for PCT Application No. PCT/US2019/068766, mailed on Apr. 22, 2020, 13 pgs.
[cited by applicant]
Long, et al.; “Fully convolutional networks for semantic segmentation”, CVPR, Nov. 2015, 10 pgs.
[cited by applicant]
Alvarez, et al.; “Road scene segmentation from a single image”, In Proceedings of the 12th European Conference on Computer Vision—vol. Part VII, ECCV'12, pp. 376-389, Berlin, Heidelberg, 2012.
[cited by applicant]
Brust, et al.; “Convolutional Patch networks with spatial prior for road detection and urban scene understanding”, In International Conference on Computer Vision Theory and Applications (VISAPP), 2015.
[cited by applicant]
Mohan, Rahul; “Deep deconvolutional networks for scene parsing”, CoRR, abs/1411.4101,2014.
[cited by applicant]
Oliveira, et al.; “Efficient Deep Models for Monocular Road Segmentation”, IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2016).
[cited by applicant]
Teichmann, et al.; “Multinet: Real-time joint semantic reasoning for autonomous driving”, arXiv preprint arXiv:1612.07695, 2016.
[cited by applicant]
Wang, et al.; “Embedding Structured Contour and Location Prior in Siamesed Fully Convolutional Networks for Road Detection”, in IEEE Transactions on Intelligent Transportation Systems, vol. 19, No. 1, pp. 230-241, Jan. …
[cited by applicant]
Mendes, et al.; “Exploiting fully convolutional neural networks for fast road detection”, Proc. IEEE Int. Conf. Robot. Auto. (ICRA), pp. 3174-3179, May 2016.
[cited by applicant]
“Neural Networks”, https://www.tensorflow.org/api_guides/python/nn#conv2d_transpose.
[cited by applicant]
Han, et al.; “Learning both Weights and Connections for Efficient Neural Networks”, https://arxiv.org/abs/1506.02626.
[cited by applicant]
Molchanov, et al.; “Pruning Convolutional Neural Networks for Resource Efficient Inference”, https://arxiv.org/ abs/1611.06440.
[cited by applicant]
Elfes; “Sonar-based real-world mapping and navigation”, Journal of Robotics and Automation, 1987.
[cited by applicant]
Thrun, et al.; “Probabilistic Robotics: Intelligent Robotics and Autonomous Agents” The MIT Press, 2005.
[cited by applicant]
Badino, et al.; “Free space computation using stochastic occupancy grids and dynamic programming”, In ICCV Workshop on Dynamical Vision, 2007.
[cited by applicant]
Franke, et al.; “Fast stereo based object detection for stop and go traffic”, In IV, 1996.
[cited by applicant]
Badino, et al.; “The stixel world—a compact medium level representation of the 3d-world”, in DAGM, 2009.
[cited by applicant]
Hirschmuller, “Stereo processsing by semiglobal matching and mutual information”, PAMI, 2008.
[cited by applicant]
Benenson; “Stixels estimation without depth map computation”, In ICCV, 2011.
[cited by applicant]
Yao, et al.; “Estimating drivable collision-free space from monocular video”, in Applications of Computer Vision, 2015, pp. 420-427.
[cited by applicant]
Levi, et al.; “Stixelnet: A deep convolutional network for obstacle detection and road segmentation”, 26th British Machine Vision Conference (BMVC) 2015.
[cited by applicant]
Notice of Allowance for U.S. Appl. No. 16/355,328, filed Mar. 15, 2019, mailed Apr. 20, 2022, 5 pgs.
[cited by applicant]
Notice of Allowance for U.S. Appl. No. 16/911,007, filed Jun. 24, 2020, mailed May 11, 2022, 9 pgs.
[cited by applicant]
Bouttefroy, Philippe; Notice of Allowance for U.S. Appl. No. 16/385,921, filed Apr. 16, 2019, mailed Dec. 24, 2020, 18 pgs.
[cited by applicant]
Smolyanskiy, Nikolai; Notice of Allowance for U.S. Appl. No. 16/356,439, filed Mar. 18, 2019, mailed Mar. 26, 2021, 11 pgs.
[cited by applicant]
Pham, et al.; Pre-Interview First Office Action for U.S. Appl. No. 16/911,007, filed Jun. 24, 2020, mailed Feb. 22, 2022, 2 pgs.
[cited by applicant]
Arar, et al.; U.S. Appl. No. 62/948,796, filed Dec. 16, 2019.
[cited by applicant]
Non-Final Office Action dated Aug. 18, 2021 in U.S. Appl. No. 17/004,252, 9 pgs.
[cited by applicant]
Notice of Allowance for U.S. Appl. No. 17/004,252, filed Aug. 27, 2020, mailed Jan. 13, 2022, 9 pgs.
[cited by applicant]
Arar, Nuri Murat; Restriction Requirement for U.S. Appl. No. 17/076,690, filed Oct. 21, 2020, mailed Sep. 8, 2023, 5 pgs.
[cited by applicant]
Chen, et al.; “Angular Visual Hardness”, arXiv preprint arXiv:1912.02279v3, Feb. 27, 2020, 27 pgs.
[cited by applicant]
Murat Arar, Nuri; Notice of Allowance for U.S. Appl. No. 17/076,690, filed Oct. 21, 2020, mailed Jan. 24, 2024, 58 pgs.
[cited by applicant]
IEC 61508, “Functional Safety of Electrical/Electronic/Programmable Electronic Safety-related Systems,” https://en.wikipedia.org/wiki/IEC_61508, accessed on Apr. 1, 2022, 7 pgs.
[cited by applicant]
ISO 26262, “Road vehicle—Functional safety,” International standard for functional safety of electronic system, https://en.wikipedia.org/wiki/ISO_26262, accessed on Sep. 13, 2021, 8 pgs.
[cited by applicant]
Pham, Trung; International Preliminary Report on Patentability received for PCT Application No. PCT/US2020/039430, mailed on Jan. 6, 2022, 12 pages.
[cited by applicant]
Pham, Trung; International Search Report and Written Opinion for PCT Patent Application No. PCT/US2020/039430, mailed on Oct. 9, 2020, 13 pages.
[cited by applicant]
Luvizon, D., Picard, D., & Tabia, H. (2020). Multi-task Deep Learning for Real-Time 3D Human Pose Estimation and Action Recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence.
[cited by applicant]
Habermann, M., Xu, W., Zollhofer, M., Pons-Moll, G., & Theobalt, C. (2020). Deepcap: Monocular human performance capture using weak supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern R…
[cited by applicant]
Final Office Action dated Dec. 17, 2021 in U.S. Appl. No. 16/355,328, 17 pgs.
[cited by applicant]
Notice of Allowance dated Mar. 4, 2022 in U.S. Appl. No. 16/355,328, 5 pgs.
[cited by applicant]
Asvadi, A., et al., “DepthCN: Vehicle Detection Using 3D-LIDAR and ConvNet”, International Conference on Intelligent Transportation Systems (ITSC), IEEE, pp. 1-6 (Oct. 16, 2017), XP033330533.
[cited by applicant]
Tateno, K., et al., “CNN-SLAM: Real-time dense monocular SLAM with learned depth prediction”, Arxiv.Org, Cornell University Library, pp. 6243-6252 (Apr. 11, 2017).
[cited by applicant]
International Preliminary Report on Patentability received for PCT Application No. PCT/US2019/068766, mailed on Jul. 8, 2021, 10 pgs.
[cited by applicant]
International Preliminary Report on Patentability received for PCT Application No. PCT/US2019/068764, mailed on Jul. 8, 2021, 12 pgs.
[cited by applicant]
First Action Interview Office Action dated Mar. 1, 2021 in U.S. Appl. No. 16/355,328, 4 pages.
[cited by applicant]
Non-Final Office Action dated Nov. 25, 2020 in U.S. Appl. No. 16/356,439, 22 pgs.
[cited by applicant]
Preinterview First Office Action dated Jan. 26, 2021 in U.S. Appl. No. 16/355,328, 5 pgs.
[cited by applicant]
International Preliminary Report on Patentability received for PCT Application No. PCT/US2019/022753, mailed on Oct. 1, 2020, 14 pgs.
[cited by applicant]
Szegedy, C., et al., “Going Deeper with Convolutions”, https://arxiv.org/abs/1409.4842; Sep. 17, 2014, 12 pgs.
[cited by applicant]
International Search Report and Written Opinion mailed Nov. 7, 2019 in International Patent Application No. PCT/US2019/022753, 22 pgs.
[cited by applicant]
Kendall, Alex et al., “End-to-End Learning of Geometry and Context for Deep Stereo Regression” ARXIV.org, Cornell University Library, 201 Olin Library Cornell University Ithaca, NY 14853, Mar. 13, 2017, 10 pgs.
[cited by applicant]
International Search Report and Written Opinion mailed Jul. 25, 2019 in International Patent Application No. PCT/US2019/018348, 22 pgs.
[cited by applicant]
International Search Report and Written Opinion mailed Jun. 26, 2019 in International Patent Application No. PCT/US2019/024400, 15 pgs.
[cited by applicant]
International Search Report and Written Opinion mailed Aug. 26, 2019 in International Patent Application No. PCT/US2019/022592, 18 pgs.
[cited by applicant]
“Euler spiral”, Wikipedia, Retrieved from Internet URL : https://en.wikipedia.org/wiki/Euler_spiral, accessed on Feb. 21, 2019, pp. 10.
[cited by applicant]
“F1 score”, Wikipedia, Retrieved from Internet URL : https://en.wikipedia.org/wiki/F-score, accessed on Feb. 21, 2019, pp. 3.
[cited by applicant]
“Polynomial curve fitting”, Retrieved from Internet URL : https://www.mathworks.com/help/matlab/ref/polyfit.html, accessed on Feb. 21, 2019, pp. 13.
[cited by applicant]
“Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles”, Society of Automotive Engineers (SAE), Standard No. J3016-201609, pp. 30 (Sep. 30, 2016).
[cited by applicant]
“Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles”, Society of Automotive Engineers (SAE), Standard No. J3016-201806, pp. 35 (Jun. 15, 2018).
[cited by applicant]
“Tensorflow”, Retrieved from the Internet URL :https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/kernels/hinge-loss.h, accessed on May 16, 2019, pp. 1-4.
[cited by applicant]
“tf.losses.get_regularization_loss”, TensorFlow Core 1.13, Retrieved from the Internet URL : https://www.tensorflow.org/api_docs/python/tf/losses/get_regularization_loss, accessed on May 16, 2019, pp. 1-1.
[cited by applicant]
“tf.while_loop much slower than static graph? #9527”, tensorflow, Retrieved from the Internet URL : https://github.com/tensorflow/tensorflow/issues/9527, accessed on May 16, 2019, pp. 1-7.
[cited by applicant]
What is polyline?, Webopedia Definition, Retrieved from Internet URL : https://www.webopedia.com/TERM/P/polyline.html, accessed on Feb. 21, 2019, pp. 4.
[cited by applicant]
Bach, M., et al., “Multi-camera traffic light recognition using a classifying Labeled Multi-Bernoulli filter”, IEEE Intelligent Vehicles Symposium (IV), pp. 1045-1051 (Jun. 2017).
[cited by applicant]
Bidlack, C., et al., “Visual Robot Navigation using Flat Earth Obstacle Projection”, Proceedings of the IEEE International Conference on Robotics and Automation, pp. 3374-3381 (1994).
[cited by applicant]
Bojarski, M., et al., “End to End Learning for Self-Driving Cars”, arXiv: 1604.07316v1 [cs.CV], XP055570062, Retrieved from the Internet URL:https://nvidia.com/content/tegra/automotive/images/2016/solutions/pdf/end-to-e…
[cited by applicant]
Cheng, G., et al., “Automatic Road Detection and Centerline Extraction via Cascaded End-to-End Convolutional Neural Network”, IEEE Transactions on Geoscience and Remote Sensing vol. 55, No. 6, pp. 3322-3337 (Jun. 1, 201…
[cited by applicant]
Chilamkurthy, S., “A 2017 Guide to Semantic Segmentation with Deep Learning”, Qure.ai Blog, Retrieved from Internet URL : http://blog.qure.ai/notes/semantic-segmentation-deep-learning-review, accessed on Feb. 21, 2019, …
[cited by applicant]
Deshpande, A., “A Beginner's Guide to Understanding Convolutional Neural Networks”, accessed at: https://adeshpande3.github.io/A-Beginner's-Guide-To-Understanding-Convolutional-Neural-Networks/, Accessed on Feb. 21, 201…
[cited by applicant]
Garnett, N., et al., “Real-Time Category-Based and General Obstacle Detection for Autonomous Driving”, IEEE International Conference On Computer Vision Workshops, pp. 198-205 (2017).
[cited by applicant]
Godard, C., et al., “Unsupervised Monocular Depth Estimation with Left-Right Consistency”, IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 270-279 (2017).
[cited by applicant]
He, L., et al., “Learning Depth from Single Images with Deep Neural Network Embedding Focal Length”, Cornell University Library, pp. 1-14 (Mar. 27, 2018).
[cited by applicant]
Huval, B. et al., “An Empirical Evaluation of Deep Learning on Highway Driving”, Apr. 17, 2015, 7 pages. Available at: https://arxiv.org/pdf/1504.01716.pdf.
[cited by applicant]
Ioffe, S., and Szegedy, C., “Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift”, arXiv:1502.03167v3 [cs.LG], pp. 1-12 (Mar. 2, 2015), Available at: https://arxiv.org/abs/1502.0…
[cited by applicant]
Jayaraman, A., et al., “Creating 3D Virtual Driving Environments for Simulation-Aided Development of Autonomous Driving and Active Safety”, SAE Technical Paper, pp. 1-6 (2017).
[cited by applicant]
John, V., et al., “Real-time road surface and semantic lane estimation using deep features”, Signal, Image and Video Processing, vol. 12, pp. 1133-1140 (Mar. 8, 2018).
[cited by applicant]
Kim, W., S., et al., “Depth Map Coding with Distortion Estimation of Rendered View”, Proceedings of Spie, vol. 7543, pp. 75430B1-75430B10, (2010).
[cited by applicant]
Kingma, D. P., and Ba, J. L., “Adam: A Method for Stochastic Optimization”, published as a conference paper at CLR 2015, arXiv:1412.6980v9 [cs.LG], pp. 1-15 (Jan. 30, 2017).
[cited by applicant]
Kokkinos, I., “Pushing the Boundaries of Boundary Detection using Deep Learning”, Retrieved from the Internet: URL:http://arxiv.org/pdf/1511.07386v2.pdf, pp. 1-12 (2016).
[cited by applicant]
Kunze, L., et al., “Reading between the Lanes: Road Layout Reconstruction from Partially Segmented Scenes”, 2018 21st International Conference on Intelligent Transportation Systems (ITSC), pp. 401-408 (Nov. 4-7, 2018).
[cited by applicant]
Liu, H., et al., “Neural Person Search Machines”, IEEE International Conference on Computer Vision (ICCV), pp. 493-501 (2017).
[cited by applicant]
Neven, D., et al., “Towards end-to-end lane detection: an instance segmentation approach”, In 2018 IEEE intelligent vehicles symposium (IV), pp. 7 (2018).
[cited by applicant]
Pang, J., et al., “Cascade Residual Learning: A Two-Stage Convolutional Neural Network for Stereo Matching”, IEEE International Conference on Computer Vision Workshops, pp. 887-895 (2017).
[cited by applicant]
Rothe, R., et al., “Non-maximum Suppression for Object Detection by Passing Messages Between Windows”, ETH Library, pp. 1-17 (2015).
[cited by applicant]
Stein, G. P., et al., “Vision-Based ACC With a Single Camera: Bounds on Range and Range Rate Accuracy”, Proceedings of IEEE Intelligent Vehicle Symposium, pp. 1-6 (2003).
[cited by applicant]
Suorsa, R., E., and Sridhar, B., “A Parallel Implementation of a Multisensor Feature-Based Range-Estimation Method”, IEEE Transactions on Robotics and Automation, pp. 1-34 (1993).
[cited by applicant]
Tao, A., “Detectnet: Deep neural network for object detection in digits”, NVIDIA Developer Blog, Retrieved from Internet URL: https://devblogs.nvidia.com/detectnet-deep-neural-network-object-detection-digits/, accessed …
[cited by applicant]
International Search Report and Written Opinion received for PCT Patent Application No. PCT/US2019/019656, mailed on Jul. 24, 2019, 14 pages.
[cited by applicant]
International Search Report and Written Opinion received for PCT Patent Application No. PCT/US2019/042225, mailed on Oct. 18, 2019, 11 pages.
[cited by applicant]
International Preliminary Report on Patentability received for PCT Application No. PCT/US2019/018348, mailed on Aug. 27, 2020, 16 pages.
[cited by applicant]
International Preliminary Report on Patentability received for PCT Application No. PCT/US2019/019656, mailed on Sep. 3, 2020, 11 pages.
[cited by applicant]
International Preliminary Report on Patentability received for PCT Application No. PCT/US2019/022592, mailed on Sep. 24, 2020, 11 pages.
[cited by applicant]
International Preliminary Report on Patentability received for PCT Application No. PCT/US2019/024400, mailed on Oct. 8, 2020, 10 pages.
[cited by applicant]
Notice of Allowance dated Jan. 4, 2021 in U.S. Appl. No. 16/535,440, 10 pages.
[cited by applicant]
Notice of Allowance dated Jan. 19, 2021 in U.S. Appl. No. 16/286,329, 8 pages.
[cited by applicant]
International Preliminary Report on Patentability received for PCT Patent Application No. PCT/US2019/042225, mailed on Jan. 28, 2021, 9 pages.
[cited by applicant]
International Search Report and Written Opinion received for PCT Patent Application No. PCT/US2020/062869, mailed on Mar. 17, 2021, 11 pages.
[cited by applicant]
Virgo, M., “Lane Detection with Deep Learning (Part 1)”, Accessed on Feb. 22, 2019 at: https://towardsdatascience.com/lane-detection-with-deep-learning-part-1-9e096f3320b7, pp. 1-10 (May 9, 2017).
[cited by applicant]
Weber, M., et al., “DeepTLR: A single deep convolutional network for detection and classification of traffic lights”, IEEE Intelligent Vehicles Symposium (IV), pp. 8 (Jun. 2016).
[cited by applicant]
Xie, S., and Tu, Z., “Holistically -Nested Edge Detection”, Computer Vision Foundation, pp. 1395-1403 (2015).
[cited by applicant]
Yang, Z., “Research on Lane Recognition Algorithm Based on Deep Learning”, International Conference on Artificial Intelligence and Advanced Manufacturing (AIAM), IEEE, pp. 387-391 (2019).
[cited by applicant]
Zhong, Y., et al., “Self-Supervised Learning for Stereo Matching with Self-Improving Ability”, Cornell University Library, pp. 1-13 (2017).
[cited by applicant]