Dense three-dimensional correspondence estimation with multi-level metric learning and hierarchical matching
A method for estimating dense 3D geometric correspondences between two input point clouds by employing a 3D convolutional neural network (CNN) architecture is presented. The method includes, during a training phase, transforming the two input point clouds into truncated distance function voxel grid representations, feeding the truncated distance function voxel grid representations into individual feature extraction layers with tied weights, extracting low-level features from a first feature extraction layer, extracting high-level features from a second feature extraction layer, normalizing the extracted low-level features and high-level features, and applying deep supervision of multiple contrastive losses and multiple hard negative mining modules at the first and second feature extraction layers. The method further includes, during a testing phase, employing the high-level features capturing high-level semantic information to obtain coarse matching locations, and refining the coarse matching locations with the low-level features to capture low-level geometric information for estimating precise matching locations.
1. A computer-implemented method executed on a processor for estimating dense three-dimensional (3D) geometric correspondences between two input point clouds by employing a 3D convolutional neural network (CNN) architecture, the method comprising:
during a training phase:
transforming the two input point clouds into truncated distance function voxel grid representations;
feeding the truncated distance function voxel grid representations into individual feature extraction layers with tied weights;
extracting low-level features from a first feature extraction layer;
extracting high-level features from a second feature extraction layer;
normalizing the extracted low-level features and high-level features to obtain unit vector features; and
applying deep supervision of multiple contrastive losses and multiple hard negative mining modules at the first and second feature extraction layers; and
during a testing phase:
employing the high-level features capturing high-level semantic information to obtain coarse matching locations; and
refining the coarse matching locations with the low-level features to capture low-level geometric information for estimating precise matching locations.
2. The method of claim 1 , further comprising applying deep supervision of multiple contrastive losses to multiple layers of 3D CNN architecture for descriptor learning.
3. The method of claim 1 , further comprising employing a hierarchical matching strategy within activation maps of the 3D CNN architecture to fuse the high-level semantic information and the low-level geometric information.
4. The method of claim 3 , wherein the high-level semantic information includes at least object class and object pose.
5. The method of claim 3 , wherein the low-level geometric information includes at least color and edge data.
6. The method of claim 1 , wherein the 3D geometric correspondences are fed into a point cloud registration device to compute either a rigid transformation or a non-rigid transformation.
7. The method of claim 6 , wherein the point cloud registration device outputs an aligned point cloud providing a larger view of a scene.
8. A non-transitory computer-readable storage medium comprising a computer-readable program for estimating dense three-dimensional (3D) geometric correspondences between two input point clouds by employing a 3D convolutional neural network (CNN) architecture, wherein the computer-readable program when executed on a computer causes the computer to perform the steps of:
during a training phase:
transforming the two input point clouds into truncated distance function voxel grid representations;
feeding the truncated distance function voxel grid representations into individual feature extraction layers with tied weights;
extracting low-level features from a first feature extraction layer;
extracting high-level features from a second feature extraction layer;
normalizing the extracted low-level features and high-level features to obtain unit vector features; and
applying deep supervision of multiple contrastive losses and multiple hard negative mining modules at the first and second feature extraction layers; and
during a testing phase:
employing the high-level features capturing high-level semantic information to obtain coarse matching locations; and
refining the coarse matching locations with the low-level features to capture low-level geometric information for estimating precise matching locations.
9. The non-transitory computer-readable storage medium of claim 8 , wherein deep supervision of multiple contrastive losses is applied to multiple layers of 3D CNN architecture for descriptor learning.
10. The non-transitory computer-readable storage medium of claim 8 , wherein a hierarchical matching strategy is employed within activation maps of the 3D CNN architecture to fuse the high-level semantic information and the low-level geometric information.
11. The non-transitory computer-readable storage medium of claim 10 , wherein the high-level semantic information includes at least object class and object pose.
12. The non-transitory computer-readable storage medium of claim 10 , wherein the low-level geometric information includes at least color and edge data.
13. The non-transitory computer-readable storage medium of claim 8 , wherein the 3D geometric correspondences are fed into a point cloud registration device to compute either a rigid transformation or a non-rigid transformation.
14. The non-transitory computer-readable storage medium of claim 13 , wherein the point cloud registration device outputs an aligned point cloud providing a larger view of a scene.
15. A system for estimating dense three-dimensional (3D) geometric correspondences between two input point clouds by employing a 3D convolutional neural network (CNN) architecture, the system comprising:
a training module for training by:
transforming the two input point clouds into truncated distance function voxel grid representations;
feeding the truncated distance function voxel grid representations into individual feature extraction layers with tied weights;
extracting low-level features from a first feature extraction layer;
extracting high-level features from a second feature extraction layer;
normalizing the extracted low-level features and high-level features to obtain unit vector features; and
applying deep supervision of multiple contrastive losses and multiple hard negative mining modules at the first and second feature extraction layers; and
a testing module for testing by:
employing the high-level features capturing high-level semantic information to obtain coarse matching locations; and
refining the coarse matching locations with the low-level features to capture low-level geometric information for estimating precise matching locations.
16. The system of claim 15 , wherein deep supervision of multiple contrastive losses is applied to multiple layers of 3D CNN architecture for descriptor learning.
17. The system of claim 15 , wherein a hierarchical matching strategy is employed within activation maps of the 3D CNN architecture to fuse the high-level semantic information and the low-level geometric information.
18. The system of claim 17 , wherein the high-level semantic information includes at least object class and object pose, and the low-level geometric information includes at least color and edge data.
19. The system of claim 15 , wherein the 3D geometric correspondences are fed into a point cloud registration device to compute either a rigid transformation or a non-rigid transformation.
20. The system of claim 19 , wherein the point cloud registration device outputs an aligned point cloud providing a larger view of a scene.