IP Library › Granted Patent US 12,243,185
Granted Patent B2
US 12,243,185 · App. 17/852,636 · Granted Mar 4, 2025

Systems and methods for point cloud registration

Inventors: Minghan Zhu (Ann Arbor, MI); Maani Ghaffari (Ann Arbor, MI); Huei Peng (Ann Arbor, MI)
Assignee: THE REGENTS OF THE UNIVERSITY OF MICHIGAN
G06T3/14G06T7/344G06T2207/10028G06T2207/20084G06T2207/30252
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,243,185
App. No.
17/852,636
Granted
Mar 4, 2025
Kind
B2
Abstract

Systems and methods are provided for point cloud processing with an equivariant neural network and implicit shape learning that may produce correspondence-free registration. The systems and methods may provide for feature space preservation with the same rotation operation as a Euclidean input space, due to the equivariance property, which may provide for solving the feature-space registration in a closed form.

Claims (107)

1. A method for point cloud registration, comprising:

accessing point cloud input data that includes at least a first point cloud and a second point cloud,

wherein the second point cloud is rotated from the first point cloud about an origin;

subjecting the point cloud input data to an equivariant encoder to extract features from the point cloud input data;

performing shape reconstruction by subjecting the extracted features to a decoder;

performing registration in feature space by aligning the decoded features for the first point cloud and second point cloud;

determining an optimized rotation matrix to translate between the first point cloud and the second point cloud.

2. The method of claim 1 , wherein the point cloud registration is correspondence-free and includes SO(3)-equivariance representation learning with a neural network.

3. The method of claim 1 , wherein the point cloud registration generates a closed-form solution.

4. The method of claim 1 , further comprising determining an occupancy value prediction.

5. The method of claim 4 , wherein occupancy value prediction includes where the decoder uses an encoded shape feature and a queried position in a feature space.

6. The method of claim 1 , further comprising determining a loss in the shape reconstruction given by:

L

OCC

=

∑

i

=

1

n

L

cross

⁢

_

⁢

entropy

(

v

⁡

(

p

i

;

f

⁡

(

P

)

)

,

v

gt

(

p

i

)

)

,

where L occ represents occupancy value loss, v gt (p)∈{0; 1} represents a ground truth occupancy value of a query point p; f(P 1 ); f(P 2 ), represents a feature of the point clouds P 1 ; P 2 .

7. The method of claim 1 , wherein the optimized rotation matrix is determined using a singular value decomposition (SVD).

8. The method of claim 1 , wherein the point cloud registration is independent of at least one of initial rotation error, input point clouds, noise, sampling, and density differences.

9. The method of claim 1 , wherein the point cloud registration is performed using a neural network.

10. The method of claim 1 , wherein the point cloud registration is performed by a vehicle system to control autonomous driving.

11. A system for point cloud registration, comprising:

a computer system configured to:

i) access point cloud input data that includes at least a first point cloud and a second point cloud,

wherein the second point cloud is rotated from the first point cloud about an origin;

ii) subject the point cloud input data to an equivariant encoder to extract features from the point cloud input data;

iii) perform shape reconstruction by subjecting the extracted features to a decoder;

iv) perform registration in feature space by aligning the decoded features for the first point cloud and second point cloud;

v) determine an optimized rotation matrix to translate between the first point cloud and the second point cloud.

12. The system of claim 11 , wherein the point cloud registration is correspondence-free and includes SO(3)-equivariance representation learning with a neural network.

13. The system of claim 11 , wherein the point cloud registration generates a closed-form solution.

14. The system of claim 11 , wherein the computer system is further configured to determine an occupancy value prediction.

15. The system of claim 14 , wherein occupancy value prediction includes where the decoder uses an encoded shape feature and a queried position in a feature space.

16. The system of claim 11 , wherein the computer system is further configured to determine a loss in the shape reconstruction given by:

L

OCC

=

∑

i

=

1

n

L

cross

⁢

_

⁢

entropy

(

v

⁡

(

p

i

;

f

⁡

(

P

)

)

,

v

gt

(

p

i

)

)

,

where L occ represents occupancy value loss, v gt (p)∈{0; 1} represents a ground truth occupancy value of a query point p; f(P 1 ); f(P 2 ), represents a feature of the point clouds P 1 ; P 2 .

17. The system of claim 11 , wherein the computer system is further configured to determine the optimized rotation matrix using a singular value decomposition (SVD).

18. The system of claim 11 , wherein the point cloud registration is independent of at least one of initial rotation error, input point clouds, noise, sampling, and density differences.

19. The system of claim 11 , wherein the computer system is further configured to perform point cloud registration using a neural network.

20. The system of claim 11 , wherein the computer system is further configured to perform point cloud registration with a vehicle system to control autonomous driving.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 2, 2023
From: ZHU, MINGHAN; GHAFFARI, MAANI; PENG, HUEI
To: MICHIGAN, THE REGENTS OF THE UNIVERSITY OF
Reel/Frame 064468/0261 →
Continuity (2)
Provisional Application 63216042 · Jun 29, 2021
Related Publication 20220414821A1 · Dec 29, 2022
References Cited (49)
US 20220084221A1 · Deng · 2022 [cited by examiner]
Spezialetti et al., “Learning to orient surfaces by self-supervised spherical CNNs” (Year: 2020). [cited by examiner]
Ao, S. et al., SpinNet: Learning a General Surface Descriptor for 3D Point Cloud Registration, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 11753-11762. [cited by applicant]
Aoki, Y. et al., PointNetLK: Robust & Efficient Point Cloud Registration Using Pointnet, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 7163-7172. [cited by applicant]
Bai, Y. et al., D3Feat: Joint Learning of Dense Detection and Description of 3D Local Features, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 6359-6367. [cited by applicant]
Besl, P. et al., A Method for Registration of 3-D Shapes, IEEE Transactions on Pattern Analysis and Machine Intelligence, 1992, 14(2):239-256. [cited by applicant]
Censi, A., An ICP Variant Using a Point-to-Line Metric, In 2008 IEEE International Conference on Robotics and Automation, 2008, pp. 19-25. [cited by applicant]
Chen, Y. et al., Object Modelling by Registration of Multiple Range Images, Image and Vision Computing, 1992, 10(3):145-155. [cited by applicant]
Choy, C. et al., Fully Convolutional Geometric Features, In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 8958-8966. [cited by applicant]
Choy, C. et al., Deep Global Registration, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 2514-2523. [cited by applicant]
Cohen, T. et al., Learning the Irreducible Representations of Commutative Lie Groups, In International Conference on Machine Learning, PMLR, 2014, pp. 1755-1763. [cited by applicant]
Cohen, T. et al., Steerable CNNs, arXiv: 1612.08498, 2016, 14 pages. [cited by applicant]
Deng, H. et al., PPFNnet: Global Context Aware Local Features for Robust 3D Point Matching, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 195-205. [cited by applicant]
Deng, C. et al., Vector Neurons: A General Framework for SO(3)-Equivariant Networks, In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, p. 12200-12209. [cited by applicant]
Esteves, C. et al., Learning SO(3) Equivariant Representations with Spherical CNNs, In Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 52-68. [cited by applicant]
Finzi, M. et al., Generalizing Convolutional Neural Networks for Equivariance to Lie Groups on Arbitrary Continuous Data, in International Conference on Machine Learning, PMLR, 2020, pp. 3165-3176. [cited by applicant]
Fuchs, F. et al., SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks, Advances in Neural Information Processing Systems, 2020, 33:1970-1981. [cited by applicant]
Ghaffari, M. et al., Continuous Direct Sparse Visual Odometry from RGB-D Images, Robotics: Science and Systems, 2019, 9 pages. [cited by applicant]
Gojcic, Z. et al., The Perfect Match: 3D Point Cloud Matching with Smoothed Densities, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 5545-5554. [cited by applicant]
Gold, S. et al., New Algorithms for 2D and 3D Point Matching: Pose Estimation and Correspondence, Pattern Recognition, 1998, 31(8):1019-1031. [cited by applicant]
Granger, S. et al., Multi-scale EM-ICP: A Fast and Robust Approach for Surface Registration, In Computer Vision—ECCV 2002: 7th European Conference on Computer Vision, 2002, pp. 418-432. [cited by applicant]
Horn, B. et al., Closed-Form Solution of Absolute Orientation Using Orthonormal Matrices, Journal of the Optical Society of America A, 1988, 5(7):1127-1135. [cited by applicant]
Huang, X. et al., Feature-Metric Registration: A Fast Semi-supervised Approach for Robust Point Cloud Registration without Correspondences, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogn… [cited by applicant]
Huang, J. et al., DI-Fusion: Online Implicit 3D Reconstruction with Deep Priors, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 8932-8941. [cited by applicant]
Kondor, R. et al., On the Generalization of Equivariance and Convolution in Neural Networks to the Action of Compact Groups, In International Conference on Machine Learning, PMLR, 2018, pp. 2747-2755. [cited by applicant]
Lang, I. et al., SampleNet: Differentiable Point Cloud Sampling, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 7578-7588. [cited by applicant]
Li, J. et al., USIP: Unsupervised Stable Interest Point Detection from 3D Point Clouds, In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 361-370. 2019. [cited by applicant]
Lu, W. et al., DeepVCP: An End-to-End Deep Neural Network for Point Cloud Registration, In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 12-21. [cited by applicant]
Mescheder, L. et al., Occupancy Networks: Learning 3D Reconstruction in Function Space, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 4460-4470. [cited by applicant]
Mildenhall, B. et al., NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis, arXiv:2003.08934, 2020, 25 pages. [cited by applicant]
Mitra, N. et al., Registration of Point Cloud Data from a Geometric Optimization Perspective, In Proceedings of the 2004 Eurographics/ACM SIGGRAPH Symposium on Geometry Processing, 2004, pp. 22-31. [cited by applicant]
Pais, G. et al., 3DRegNet: A Deep Neural Network for 3D Point Registration, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 7193-7203. [cited by applicant]
Park, J. et al., DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 165-174. [cited by applicant]
Qi, C. et al., PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 652-660. [cited by applicant]
Sarode, V. et al., PCRNet: Point Cloud Registration Network Using PointNet Encoding, arXiv preprint arXiv:1908.07906, 2019, pp. 1-10. [cited by applicant]
Segal, A. et al., Generalized-ICP, In Robotics: Science and Systems, 2009, 2(4):435, 8 pages. [cited by applicant]
Shotton, J. et al., Scene Coordinate Regression Forests for Camera Relocalization in RGB-D Images, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2013, pp. 2930-2937. [cited by applicant]
Stutz, D. et al., Learning 3D Shape Completion Under Weak Supervision, International Journal of Computer Vision, 2020, 128:1162-1181. [cited by applicant]
Wang, Y. et al., Deep Closest Point: Learning Representations for Point Cloud Registration, In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 3523-3532. [cited by applicant]
Wang, Y. et al., Dynamic Graph CNN for Learning on Point Clouds, ACM Transactions on Graphics, 2019, vol. 38, No. 5, Article 146, pp. 1-12. [cited by applicant]
Wang, Y. et al., PRNet: Self-Supervised Learning for Partial-to-Partial Registration, Advances in Neural Information Processing Systems, 2019, 32:8814-8826. [cited by applicant]
Wedderburn, R., Quasi-likelihood Functions, Generalized Linear Models, and the Gauss-Newton Method, Biometrika, 1974, 61(3):439-447. [cited by applicant]
Wu, Z. et al., 3D ShapeNets: A Deep Representation for Volumetric Shapes, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 1912-1920. [cited by applicant]
Yen-Chen, L. et al., iNeRF: Inverting Neural Radiance Fields for Pose Estimation, arXIV:2012.05877, 2021, 10 pages. [cited by applicant]
Yew, Z. et al., 3DFeat-Net: Weakly Supervised Local 3D Features for Point Cloud Registration, In Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 607-623. [cited by applicant]
Yew, Z. et al., RPM-Net: Robust Point Matching Using Learned Features, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 11824-11833. [cited by applicant]
Yuan, W. et al., DeepGMR: Learning Latent Gaussian Mixture Models for Registration, In Computer Vision-ECCV, 2020, pp. 733-750. [cited by applicant]
Zeng, A. et al., 3DMatch: Learning Local Geometric Descriptors from RGB-D Reconstructions, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 1802-1811. [cited by applicant]
Zhang, R. et al., A New Framework for Registration of Semantic Point Clouds from Stereo and RGB-D Cameras, arXiv:2012.03683, 2020, 8 pages. [cited by applicant]