Method for determining an image descriptor, encoding pipeline, and visual place recognition method
In order to improve visual place recognition, a computer implemented method for determining an image descriptor (V(X)) is provided. The method comprises the steps of: a) clustering features from a feature map ( 18 ) into a plurality of feature clusters by determining for each feature an inter-cluster weight (α); b) for each feature cluster obtained in step a), determining and assigning an intra-cluster saliency weight (β) to each feature that is associated with said feature cluster; and c) determining the image descriptor (V(X)) based on the inter-cluster weight (α) and the intra-cluster saliency weight (β).
1 . A computer implemented method for determining an image descriptor (V(X)) for use in visual place recognition, the method comprising:
a) clustering features from a feature map ( 18 ) into a plurality of feature clusters by determining for each feature an inter-cluster weight (α);
b) for each feature cluster obtained in step a), determining and assigning an intra-cluster saliency weight (β) to each feature that is associated with said feature cluster; and
c) determining the image descriptor (V(X)) based on the inter-cluster weight (α) and the intra-cluster saliency weight (β).
2 . The method according to claim 1 , characterized in that, in step a) the features to be clustered are extracted from an input image ( 20 ) by means of a convolutional neural network ( 22 ).
3 . The method according to claim 1 , characterized in that, in step a) the features are clustered using a partitional clustering method, a k-means clustering method, and/or a soft-assignment.
4 . The method according to claim 1 , characterized in that, in step b) a Voronoi cell of at least one feature cluster is separated into an informative portion, that is represented by a representative centroid (cr) that is representative of said cluster, and at least one ambiguous portion, that is represented by a shadow centroid (cs).
5 . The method according to claim 4 , characterized in that, the intra-cluster saliency weight (β) is determined such that the intra-cluster saliency weight (β) indicates a probability of a feature being located in the informative portion.
6 . The method according to claim 4 , characterized in that, in step b) a feature determined to be located in the ambiguous portion is assigned a lower intra-cluster saliency weight (β) compared to a feature determined to be located in the informative portion.
7 . The method according to claim 4 , characterized in that, in step b), for each feature cluster, the at least one shadow centroid (cs) is initialized based on at least one top candidate selected from the at least one ambiguous portion that has the closest Euclidean distance from the representative centroid (cr).
8 . The method according to claim 1 , characterized in that, in step c) an attentional pyramid pooling module ( 36 ) is used for highlighting local residuals of informative portions, and the feature map ( 18 ) is divided into a plurality of multi-scale regions by a pyramid structure having at least one pyramid level, wherein at least one regional feature (fkn,m) of the pyramid level is determined by aggregating salient cluster-wise residuals within a spatial grid that are weighted with the inter-cluster weight (α) and the intra-cluster saliency weight (β).
9 . The method according to claim 8 , characterized in that, a contribution of each regional feature (fkn,m) to a feature embedding of each feature cluster is determined according to a relative spatial saliency value that is determined based on a distinctiveness value (μkn,m) calculated by a convolutional layer that is configured to evaluate a distinctiveness of each spatial region.
10 . The method according to claim 9 , characterized in that, for each feature cluster a visual word vector (Vk) is generated from the feature embedding by aggregating the regional features (fkn,m) weighted by their respective spatial saliency value (μkn,m).
11 . The method according to claim 10 , characterized in that, the image descriptor (V(X)) is generated by normalizing the visual word vector (Vk) for each feature cluster, rescaling each normalized visual word vector with training weights (γ), and concatenating the rescaled normalized visual word vectors to form the image descriptor (V(X)).
12 . An encoding pipeline ( 10 ) for determining an image descriptor (V(X)) from an input image ( 20 ), the encoding pipeline ( 10 ) comprising a feature extraction module ( 12 ) configured to perform step a) of the method according to claim 1 , a local refinement module ( 14 ) that is configured for performing step b) of the method according to claim 1 , and a global integration module ( 16 ) configured for performing step c) of the method according to claim 1 .
13 . A method for visual place recognition, the method comprising:
a) recording an image of a scene and determining a feature map ( 18 ) of the scene;
b) performing a method according to claim 1 with the feature map ( 18 ) of the scene, so as to obtain an image descriptor (V(X)); and
c) comparing the image descriptor (V(X)) with reference images stored in a data storage and, in response to an image matching the image descriptor (V(X)) being found, generating a control signal that causes another device.
14 . A vehicle, comprising:
an image sensor configured for recording an image of a scene; and
a visual place recognition device that is configured for performing the method according to claim 13 , so as to obtain a control signal; and
a control unit that is configured to process the control signal and perform an action indicated by the control signal.
15 . A non-transitory machine readable storage medium comprising instructions that, upon execution on a data processing device, cause the device to perform at least one of the steps of the method according to claim 1 .