Dense correspondence estimation with multi-level metric learning and hierarchical matching

1. A method for localization with flexible ground modeling, comprising:

communicating a plurality of two-dimensional (2D) images of an environment to a correspondence estimation module, where the 2D images include at least a first image and a second image, wherein each of the first image and the second image are captured from an image capturing device;

hierarchically extracting first features, including geometric features and semantic features, from the first image with a first convolutional neural network (CNN) according to activation map weights;

hierarchically extracting second features, including geometric features and semantic features, from the second image with a second CNN according to the activation map weights;

estimating correspondences between the first features and the second features of a terrain surface including hierarchical fusing of geometric correspondences and semantic correspondences;

generating a 3-dimensional (3D) model of a terrain using the estimated correspondences corresponding to the terrain surface;

determining relative locations of elements and objects in the environment according to the 3D model of the terrain; and

notifying a user of the relative locations.

2. The method as recited in claim 1 , further comprising updating the activation map weights, including:

mining hard negatives between the geometric features of the first image and the geometric features of the second image;

mining hard negatives between the semantic features of the first image and the semantic features of the second image; and

updating the activation map weights according to a loss function based on the hard negatives.

3. The method as recited in claim 2 , wherein the loss function includes a correspondence contrastive loss (CCL) function at multiple layers of each of the first CNN and the second CNN.

4. The method as recited in claim 1 , further including:

normalizing the geometric features and the semantic features of the first image; and

normalizing the geometric features and the semantic features of the second image.

5. The method as recited in claim 1 , further including:

estimating correspondences between the semantic features of the first image and the semantic features of the second image to generate the semantic correspondence estimates;

estimating correspondences between the geometric features of the first image and the geometric features of the second image to generate the geometric correspondence estimates; and

hierarchical fusing the geometric correspondence estimates and the semantic correspondence estimates by refining the semantic correspondence estimates with the geometric correspondence estimates.

6. The method as recited in claim 1 , further including:

performing semantic segmentation on the first image to recognize a ground surface in the first image;

performing semantic segmentation on the second image to recognize the ground surface in the second image;

filtering the correspondences with the semantic segmentation in both images to generate ground correspondences; and

determining topography of the ground surface according to the ground correspondences to generate a 3-dimensional (3D) model of the ground surface.

7. The method as recited in claim 1 , further including determining a location and movement of an element or an object in the environment according to the flexible ground model.

8. A method for flexible ground modeling, comprising:

generating a first image and a second image of an environment with an image capturing device, with each of the first image and the second image being a two-dimensional (2D) image;

hierarchically extracting first features and second features, the first features including geometric features and semantic features from the first image, and the second features including geometric features and semantic features from the second image;

estimating correspondences between the first features and the second features including hierarchical fusing of geometric correspondences and semantic correspondences;

recovering a 3-dimensional (3D) model of a ground surface using the estimated correspondences corresponding to the ground surface in the environment for flexible ground modeling, including:

performing semantic segmentation on the first image to recognize the ground surface in the first image,

performing semantic segmentation on the second image to recognize the ground surface in the second image,

filtering the correspondences with the semantic segmentation in both images to generate ground correspondences, and

determining topography of the ground surface according to the ground correspondences to generate a 3-dimensional (3D) model of the ground surface for flexible ground modeling; and

displaying the 3D model as a 3D depiction of the environment.

9. The method as recited in claim 8 , further including:

hierarchically extracting the first features, including geometric features and semantic features, from the first image with a first convolutional neural network (CNN) according to activation map weights; and

hierarchically extracting the second features, including geometric features and semantic features, from the second image with a second CNN according to the activation map weights.

10. The method as recited in claim 9 , further comprising updating the activation map weights, including:

mining hard negatives between the geometric features of the first image and the geometric features of the second image;

mining hard negatives between the semantic features of the first image and the semantic features of the second image; and

updating the activation map weights according a loss function based on the hard negatives.

11. The method as recited in claim 10 , wherein the loss function includes a correspondence contrastive loss (CCL) function at multiple layers of each of the first CNN and the second CNN.

12. The method as recited in claim 8 , further including:

normalizing the geometric features and the semantic features of the first image; and

normalizing the geometric features and the semantic features of the second image.

13. The method as recited in claim 8 , further including:

estimating correspondences between the semantic features of the first image and the semantic features of the second image to generate the semantic correspondence estimates;

estimating correspondences between the geometric features of the first image and the geometric features of the second image to generate the geometric correspondence estimates; and

hierarchically fusing the geometric correspondence estimates and the semantic correspondence estimates by refining the semantic correspondence estimates with the geometric correspondence estimates.

14. The method as recited in claim 8 , further including further including determining a location of an actual location of an object according to the flexible ground model.

15. A system for localization with flexible ground modeling, comprising:

an imaging device disposed in an environment to capture a plurality of two-dimensional (2-D) images of the environment;

a dense correspondence estimator, including a processor, to receive the plurality of 2D images, where the 2D images include at least a first image and a second image, the dense correspondence estimator including:

a memory device including a first convolutional neural network (CNN) with activation map weights to hierarchically extract first features, including geometric features and semantic features, from the first image;

the memory device including a second CNN with activation map weights to hierarchically extract second features, including geometric features and semantic features, from the second image;

the processor in communication with the memory device to estimate correspondences between the first features and the second features including hierarchical fusing of geometric correspondences and semantic correspondences;

a localization module included with the memory device for generating a 3-dimensional (3D) model of a terrain using the estimated correspondences corresponding to a terrain surface and determining relative locations of elements and objects in the environment according to the 3D model of the terrain; and

a display in communication with the localization module for notifying a user of the relative locations.

16. The system as recited in claim 15 , the processor further configured to update the activation map weights, including:

mining hard negatives between the geometric features of the first image and the geometric features of the second image;

mining hard negatives between the semantic features of the first image and the semantic features of the second image; and

updating the activation map weights according to a loss function based on the hard negatives.

17. The system as recited in claim 16 , wherein the loss function includes a correspondence contrastive loss (CCL) function at multiple layers of each of the first CNN and the second CNN.

18. The system as recited in claim 15 , wherein estimating the correspondences includes:

estimating correspondences between the semantic features of the first image and the semantic features of the second image to generate the semantic correspondence estimates; and

estimating correspondences between the geometric features of the first image and the geometric features of the second image to generate the geometric correspondence estimates.

19. The system as recited in claim 15 , further including a matching unit included in the memory for hierarchical fusing the geometric correspondence estimates and the semantic correspondence estimates by refining the semantic correspondence estimates with the geometric correspondence estimates.

20. The system as recited in claim 15 , the localization module further configured to:

perform semantic segmentation on the first image to recognize the ground surface in the first image;

perform semantic segmentation on the second image to recognize the ground surface in the second image;

filter the correspondences with the semantic segmentation in both images to generate ground correspondences; and

determine topography of the ground surface according to the ground correspondences to generate a 3-dimensional (3D) model of the ground surface.