IP Library › Granted Patent US 12,307,579
Granted Patent B2
US 12,307,579 · App. 18/482,463 · Granted May 20, 2025

Deep geometric model fitting

Inventors: Rene Ranftl (Munich, DE); Vladlen Koltun (Santa Clara, CA)
Assignee: Intel Corporation
G06T15/10G06N3/045G06N20/00G06T1/20G06T3/08G06T7/20G06T7/593G06T17/10G06N3/084G06T2207/20076G06T2207/20081G06T2207/20084G06T2207/30248
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,307,579
App. No.
18/482,463
Granted
May 20, 2025
Kind
B2
Abstract

Systems, apparatuses and methods may provide for technology that generates, by a first neural network, an initial set of model weights based on input data and iteratively generates, by a second neural network, an updated set of model weights based on residual data associated with the initial set of model weights and the input data. Additionally, the technology may output a geometric model of the input data based on the updated set of model weights. In one example, the first neural network and the second neural network reduce the dependence of the geometric model on the number of data points in the input data.

Claims (36)

1. At least one memory comprising machine-readable instructions to cause at least one processor circuit to at least:

generate, with a neural network, initial model weights based on image data from a camera in an environment;

update the initial model weights based on first data and second non-image data to determine updated model weights, the first data associated with the initial model weights, the second non-image data to specify one or more constraints; and

localize the camera in the environment based on the updated model weights.

2. The at least one memory of claim 1 , wherein the neural network is a deep neural network.

3. The at least one memory of claim 1 , wherein the camera is mounted on a vehicle.

4. The at least one memory of claim 1 , wherein one or more of the at least one processor circuit is included in a localization and mapping system.

5. The at least one memory of claim 4 , wherein the localization and mapping system is a simultaneous localization and mapping system (SLAM).

6. The at least one memory of claim 1 , wherein the updated model weights are associated with a geometric model, the geometric model associated with the environment.

7. The at least one memory of claim 1 , wherein the camera is a first camera, the image data is first image data, and the machine-readable instructions are to cause one or more of the at least one processor circuit to:

generate the initial model weights based on the first image data and second image data from a second camera in the environment; and

localize the first camera and the second camera in the environment based on the updated model weights.

8. An apparatus comprising:

interface circuitry;

machine-readable instructions; and

at least one processor circuit to be programmed by the machine-readable instructions to:

generate, with a neural network, initial model weights based on image data from a camera in an environment;

update the initial model weights based on first data and second non-image data to determine updated model weights, the first data associated with the initial model weights, the second non-image data to specify one or more constraints; and

localize the camera in the environment based on the updated model weights.

9. The apparatus of claim 8 , wherein the neural network includes a deep neural network.

10. The apparatus of claim 8 , wherein the camera is mounted on a vehicle.

11. The apparatus of claim 8 , wherein one or more of the at least one processor circuit are included in a localization and mapping system.

12. The apparatus of claim 11 , wherein the localization and mapping system is a simultaneous localization and mapping system (SLAM).

13. The apparatus of claim 8 , wherein the updated model weights are associated with a geometric model, the geometric model associated with the environment.

14. The apparatus of claim 8 , wherein the camera is a first camera, the image data is first image data, and one or more of the at least one processor circuit is to:

generate the initial model weights based on the first image data and second image data from a second camera in the environment; and

localize the first camera and the second camera in the environment based on the updated model weights.

15. A method comprising:

generating, with a neural network, initial model weights based on image data from a camera in an environment;

updating, by at least one processor circuit programmed by at least one instruction, the initial model weights based on first data and second non-image data to determine updated model weights, the first data associated with the initial model weights, the second non-image data specify one or more constraints; and

localizing the camera in the environment based on the updated model weights.

16. The method of claim 15 , wherein the neural network includes a deep neural network.

17. The method of claim 15 , wherein the camera is mounted on a vehicle.

18. The method of claim 15 , wherein one or more of the at least one processor circuit is included in a localization and mapping system.

19. The method of claim 15 , wherein the updated model weights are associated with a geometric model, the geometric model associated with the environment.

20. The method of claim 15 , wherein the camera is a first camera, the image data is first image data, the generating of the initial model weights is based on the first image data and second image data from a second camera in the environment, and further including localizing the first camera and the second camera in the environment based on the updated model weights.

Continuity (3)
Continuation 17841186 · Jun 15, 2022
Continuation 15933510 · Mar 23, 2018
Related Publication 20240161387A1 · May 16, 2024
References Cited (70)
US 5548662A · Kwon · 1996 [cited by applicant]
US 6018696A · Matsuoka et al. · 2000 [cited by applicant]
US 7213008B2 · Rising, III · 2007 [cited by applicant]
US 11095887B2 · Kim · 2021 [cited by applicant]
US 11393160B2 · Ranftl et al. · 2022 [cited by applicant]
US 11816784B2 · Ranftl et al. · 2023 [cited by applicant]
US 20130230214A1 · Arth et al. · 2013 [cited by applicant]
US 20130325775A1 · Sinyavskiy et al. · 2013 [cited by applicant]
US 20150016777A1 · Abovitz · 2015 [cited by examiner]
US 20180053056A1 · Rabinovich · 2018 [cited by examiner]
US 20180075309A1 · Sathyanarayana · 2018 [cited by examiner]
US 20180268256A1 · Di Febbo · 2018 [cited by examiner]
US 20180296281A1 · Yeung et al. · 2018 [cited by applicant]
US 20180307925A1 · Wisniowski et al. · 2018 [cited by applicant]
Axel Furlan, Free your camera: 3d indoor scene understanding from arbitrary camera motion, 2013 (month unknown), British Machine Vision Conference (BMVC), pp. 1-10, http://vision.stanford.edu/pdf/furlan13.pdf (Year: 201… [cited by examiner]
Fischler et al., “Random Sample Consensus: A paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography,” Communications of the ACM, Jun. 1981, 42 pages. [cited by applicant]
Hartley, “In Defense of the Eight-Point Algorithm,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Jun. 1997, 14 pages. [cited by applicant]
Torr et al., “The Development and Comparison of Robust Methods for Estimating the Fundamental Matrix,” International Journal of Computer Vision, Sep. 1997, 30 pages. [cited by applicant]
Zhang, “Determining the Epipolar Geometry and its Uncertainty: A Review,” International Journal of Computer Vision, Mar. 1998, 30 pages. [cited by applicant]
Hartley et al., “Multiple View Geometry in Computer Vision,” Cambridge University Press, 2000, 673 pages. [cited by applicant]
Torr et al., “Milesac: A New Robust Estimator with Application to Estimating Image Geometry,” Computer Vision and Image Understanding, Aug. 2000, 19 pages. [cited by applicant]
Torr, “Bayesian Model Estimation and Selection for Epipoplar Geometry and Generic Manifold Fitting,” International Journal of Computer Vision, Oct. 2002, 27 pages. [cited by applicant]
Lowe, “Distinctive Image Features From Scale-Invariant Keypoints,” International Journal of Computer Vision, Nov. 2004, 28 pages. [cited by applicant]
Chum et al., “Matching with PROSAC—Progressive Sample Consensus,” CVPR, Jun. 2005, 7 pages. [cited by applicant]
Li, “Consensus Set Maximization With Guaranteed Global Optimality for Robust Geometry Estimation,” ICCV 2009, Oct. 2009, 7 pages. [cited by applicant]
Grabner et al., “What Makes a Chair a Chair?” CVPR 2011, Jun. 2011, 8 pages. [cited by applicant]
Hoseinnezhad et al., “An m-estimator for High Breakdown Robust Estimation in Computer Vision,” Computer Vision and Image Understanding, Aug. 2011, 6 pages. [cited by applicant]
Rublee et al., “ORB: An Efficient Alternative to SIFT or Surf,” ICCV 2011, Nov. 2011, 8 pages. [cited by applicant]
Krizhevsky et al., “ImageNet Classification With Deep Convolutional Neural Networks,” NIPS, Jan. 2012, 9 pages. [cited by applicant]
Geiger et al., “Are We Ready for Autonomous Driving? The KITTI Version Benchmark Suite,” CVPR, Jun. 16-21, 2012, 8 pages. [cited by applicant]
Lebeda et al., “Fixing the Locally Optimized RANSAC,” BMVC 2012, Sep. 2012, 11 pages. [cited by applicant]
Maas et al., “Rectifier Nonlinearities Improve Neural Network Acoustic Models,” ICML Workshops, 2013, 6 pages. [cited by applicant]
Raguarm et al., “USAC: A Universal Framework for Random Sample Consensus,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Aug. 1, 2013, 17 pages. [cited by applicant]
Girshick et al., “Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation,” CVPR, Jun. 2014, 8 pages. [cited by applicant]
Yang et al., “Optimal Essential Matrix Estimation Via Inlier-Set Maximization,” ECCV, Sep. 2014, 16 pages. [cited by applicant]
Lin et al., “Microsoft COCO: Common Objects in Context,” ECCV, Sep. 2014, 15 pages. [cited by applicant]
Kingma et al., “A Method for Stochastic Optimization,” ICLR, May 2015, 15 pages. [cited by applicant]
Long et al., “Fully Convolutional Networks for Semantic Segmentation,” CVPR, Jun. 2015, 10 pages. [cited by applicant]
Tennakoon et al., “Robust Model Fitting Using Higher Than Minimal Subset Sampling,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Jun. 2015, 13 pages. [cited by applicant]
Mur-Artal et al., “A Versatile and Accurate Monocular SLAM System,” IEEE Transactions on Robotics 2015, Oct. 2015, 15 pages. [cited by applicant]
Kendall et al., “PoseNet: A Convolutional Network for Real-Time 6-DOF Camera Relocalization,” ICCV, Dec. 2015, 9 pages. [cited by applicant]
Agrawal et al., “Learning to See by Moving,” ICCV, Dec. 2015, 12 pages. [cited by applicant]
Ionescu et al., “Matrix Back-Propagation for Deep Networks With Structured Layers,” ICCV 2015, Dec. 2015, 9 pages. [cited by applicant]
Schonberger et al., “Structure-From-Motion Revisited,” CVPR, Jun. 2016, 10 pages. [cited by applicant]
Detone et al., “Deep Image Homography Estimation,” CoRR 2016, Jun. 2016, 6 pages. [cited by applicant]
Yi et al., “LIFT: Learned Invariant Feature Transform,” ECCV, Oct. 2016, 16 pages. [cited by applicant]
Zhou et al., “Fast Global Registration,” ECCV, Oct. 2016, 16 pages. [cited by applicant]
Chin et al., “Efficient Globally Optimal Consensus Maximisation With Tree Search,” IEEE Transactions on Pattern Anaysis and Machine Intelligence, Nov. 2016, 9 pages. [cited by applicant]
Andrychowicz et al., “Learning to Learn by Gradient Descent by Gradient Descent,” NIPS, Dec. 2016, 9 pages. [cited by applicant]
Kendall et al., “Geometric Loss Functions for Camera Pose Regression With Deep Learning,” CVPR 2017, Feb. 2017, 10 pages. [cited by applicant]
Zaheer et al., “Deep Sets,” NIPS 2017, Mar. 2017, 11 pages. [cited by applicant]
Torch Contributors, “PyTorch Documentation”, Apr. 6, 2017, 236 pages. [cited by applicant]
Savinov et al., “Quad-Networks: Unsupervised Learning to Rank for Interest Point Detection,” CVPR, Jul. 2017, 9 pages. [cited by applicant]
Knapitsch et al., “Tanks and Temples: Benchmarking Large-Scale Scene Reconstruction,” ACM Transactions on Graphics, Jul. 2017, 13 pages. [cited by applicant]
Rocco et al., “Convolutional Neural Network Architecture for Geometric Matching,” CVPR, Jul. 2017, 15 pages. [cited by applicant]
Uylanov et al., “Improved Texture Networks: Maximizing Quality and Diversity in Feed-Forward Stylization and Texture Synthesis,” CVPR, Jul. 2017, 9 pages. [cited by applicant]
Brachmann et al., “DSAC—Differentiable RANSAC for Camera Localization,” CVPR, Jul. 2017, 11 pages. [cited by applicant]
Qi et al., “Deep Learning on Point Sets for 3D Classification and Segmentation,” CVPR, Jul. 2017, 19 pages. [cited by applicant]
Vongkulbhisal et al., “Discriminative Optimization: Theory and Applications to Point Cloud Registration,” CVPR, Jul. 2017, 9 pages. [cited by applicant]
Ummenhofer et al., “DeMON: Depth and Motion network for learning monocular stereo,” CVPR, Jul. 2017, 22 pages. [cited by applicant]
Nguyen et al., “Unsupervised Deep Homography: A Fast and Robust Homography Estimation Model,” CoRR 2017, Sep. 2017, 8 pages. [cited by applicant]
Yi et al.,“Learning to Find Good Correspondences,” arXiv:1711.05971, Nov. 16, 2017, 13 pages. [cited by applicant]
Wikipedia, “Random Sample Consensus,” URL:[en.wikipedia.org/wiki/random_sample_consensus], retrieved on Feb. 8, 2018, 8 pages. [cited by applicant]
Wikipedia, “NP-Hardness”, URL:[en.wikipedia.org/wiki/NP-hardness], retrieved on Feb. 8, 2018, 3 pages. [cited by applicant]
Geiger et al., “The KITTI Vision Benchmark Suite,” The Karlsruche Institute of Technology, URL:[cvlibs.net/datasets/kitti/eval_odometry.php], retrieved on Feb. 8, 2018, 5 pages. [cited by applicant]
United States Patent and Trademark Office, “Non-Final Office Action,” issued in connection with U.S. Appl. No. 17/841,186, dated Dec. 29, 2022, 17 pages. [cited by applicant]
United States Patent and Trademark Office, “Final Office Action,” issued in connection with U.S. Appl. No. 17/841,186, dated Apr. 17, 2023, 10 pages. [cited by applicant]
United States Patent and Trademark Office, “Notice of Allowance and Fee(s) Due,” issued in connection with U.S. Appl. No. 17/841,186, dated Jul. 10, 2023, 5 pages. [cited by applicant]
United States Patent and Trademark Office, “Non-Final Office Action” issued in connection with U.S. Appl. No. 15/933,510, dated Nov. 16, 2021, 13 pages. [cited by applicant]
United States Patent and Trademark Office, “Notice of Allowance and Fee(s) Due,” issued in connection with U.S. Appl. No. 15/933,510, dated Mar. 16, 2022, 7 pages. [cited by applicant]