Method and apparatus with image correspondence
A method with image correspondence includes: acquiring a plurality of feature map pairs corresponding to outputs of a plurality of layers of a convolutional neural network (CNN) in response to an input of an input image pair; selecting a portion of feature map pairs from among the plurality of feature map pairs based on a feature of each of the plurality of feature map pairs; generating a hyper feature map pair based on the selected portion of feature map pairs; and generating a correspondence result of the input image pair based on a correlation of the hyper feature map pair.
1 . A method with image correspondence, comprising:
acquiring a plurality of feature map pairs corresponding to outputs of a plurality of layers of a convolutional neural network (CNN) in response to an input of an input image pair;
selecting a portion of feature map pairs from among the plurality of feature map pairs by determining, for each of the plurality of feature map pairs, whether to select the feature map pair based on a feature of the feature map pair;
generating a hyper feature map pair based on the selected portion of feature map pairs; and
generating a correspondence result of the input image pair based on a correlation of the hyper feature map pair.
2 . The method of claim 1 , wherein
the plurality of layers of the CNN comprises one or more intermediate layers of the CNN, and
the plurality of feature map pairs comprises one or more intermediate feature map pairs output from the one or more intermediate layers of the CNN.
3 . The method of claim 1 , wherein the acquiring of the plurality of feature map pairs comprises:
acquiring first feature maps output from the plurality of layers of the CNN by inputting a first input image of the input image pair to the CNN;
acquiring second feature maps output from the plurality of layers of the CNN by inputting a second input image of the input image pair to the CNN; and
determining the plurality of feature map pairs by pairing the first feature maps and the second feature maps according to the plurality of layers of the CNN.
4 . The method of claim 1 , wherein the selecting of the portion of feature map pairs comprises dynamically selecting the portion of feature map pairs from among the plurality of feature map pairs based on outputs of one or more neural networks determined based on each of the plurality of feature map pairs.
5 . The method of claim 4 , wherein the one or more neural networks comprise:
a first neural network configured to, for each of the plurality of feature map pairs, encode a channel-wise feature vector corresponding to the feature map pair to a relevance vector; and
a second neural network configured to, for each of the plurality of feature map pairs, transform the feature map pair to another feature map pair by reducing a channel-wise dimension of the feature map pair.
6 . The method of claim 4 , wherein the one or more neural networks are trained to select the portion of feature map pairs from among the plurality of feature map pairs based on a selection ratio determined by a configuration parameter.
7 . The method of claim 1 , wherein the selecting of the portion of feature map pairs comprises:
determining a first feature vector corresponding to a first feature map pair among the plurality of feature map pairs;
acquiring a first relevance vector corresponding to an output of a multi-layer perceptron in response to an input of the first feature vector; and
determining whether to select the first feature map pair based on a value of the first relevance vector.
8 . The method of claim 1 , wherein the selecting of the portion of feature map pairs comprises selecting the portion of feature map pairs from among the plurality of feature map pairs by processing each of the plurality of feature map pairs in parallel to consider the feature of each of the plurality of feature map pairs.
9 . The method of claim 1 , wherein the selecting of the portion of feature map pairs comprises selecting the portion of feature map pairs from among the plurality of feature map pairs based on a selection ratio determined by a configuration parameter.
10 . The method of claim 9 , wherein the configuration parameter is determined based on either one or both of a speed and an accuracy of an application.
11 . The method of claim 1 , wherein the generating of the hyper feature map pair comprises generating the hyper feature map pair by performing upsampling and concatenation based on the selected portion of feature map pairs.
12 . The method of claim 1 , wherein the generating of the hyper feature map pair comprises generating the hyper feature map pair by combining other feature map pairs corresponding to the selected portion of feature map pairs.
13 . A non-transitory computer-readable record medium storing instructions that, when executed by a processor, configure the processor to perform the method of claim 1 .
14 . An apparatus with image correspondence, comprising:
a processor configured to
acquire a plurality of feature map pairs corresponding to outputs of a plurality of layers of a convolutional neural network (CNN) in response to an input of an input image pair,
select a portion of feature map pairs from among the plurality of feature map pairs by determining, for each of the plurality of feature map pairs, whether to select the feature map pair based on a feature of the feature map pair,
generate a hyper feature map pair based on the selected portion of feature map pairs, and
generate a correspondence result of the input image pair based on a correlation of the hyper feature map pair.
15 . The apparatus of claim 14 , wherein the plurality of layers of the CNN comprises one or more intermediate layers of the CNN, and
the plurality of feature map pairs comprises one or more intermediate feature map pairs output from the one or more intermediate layers of the CNN.
16 . The apparatus of claim 14 , wherein, for the determining of the portion of feature map pairs, the processor is configured to dynamically select the portion of feature map pairs from among the plurality of feature map pairs based on outputs of one or more neural networks determined based on each of the plurality of feature map pairs.
17 . The apparatus of claim 14 , wherein, for the determining of the portion of feature map pairs, the processor is configured to select the portion of feature map pairs from among the plurality of feature map pairs based on a selection ratio determined by a configuration parameter.
18 . The apparatus of claim 14 , further comprising a memory storing instructions that, when executed by the processor, configure the processor to perform the acquiring of the plurality of feature map pairs, the determining of the portion of feature map pairs, the generating of the hyper feature map pair, and the generating of the correspondence result.
19 . The apparatus of claim 18 , wherein the apparatus is an electronic device comprising a camera configured to generate one or more input images of the input image pair.
20 . An electronic device comprising:
a camera configured to generate one or more input images of an input image pair; and
a processor configured to
acquire a plurality of feature map pairs corresponding to outputs of a plurality of layers of a convolutional neural network (CNN) in response to an input of the input image pair,
select a portion of feature map pairs from among the plurality of feature map pairs by determining, for each of the plurality of feature map pairs, whether to select the feature map pair based on a feature of the feature map pair,
generate a hyper feature map pair based on the selected portion of feature map pairs, and
generate a correspondence result of the input image pair based on a correlation of the hyper feature map pair.
21 . The device of claim 20 , wherein
the plurality of layers of the CNN comprises one or more intermediate layers of the CNN, and
the plurality of feature map pairs comprises one or more intermediate feature map pairs output from the one or more intermediate layers of the CNN.
22 . The device of claim 20 , wherein, for the determining of the portion of feature map pairs, the processor is configured to dynamically select the portion of feature map pairs from among the plurality of feature map pairs based on outputs of one or more neural networks determined based on each of the plurality of feature map pairs.
23 . A method with image correspondence, comprising:
determining, using one or more neural networks, relevance vectors of feature map pairs output from intermediate convolutional neural network (CNN) layers, wherein the feature map pairs are generated based on an image pair;
dynamically selecting, based on the relevance vectors, a portion of the feature map pairs for generating a hyper feature map pair by determining, for each of the plurality of feature map pairs, whether to select the feature map pair based on one of the relevance vectors determined for the feature map pair; and
generating a correspondence result of the image pair based on a correlation of the hyper feature map pair.
24 . The method of claim 23 , wherein the dynamically selecting comprises determining whether to select or skip a feature map pair of the feature map pairs for the generating of the hyper feature map pair, based on a value of the one of the relevance vectors determined for the feature map pair.
25 . The method of claim 23 , wherein each of the feature map pairs are output from a different one of the intermediate CNN layers, and each of the feature map pairs include a feature map corresponding to a first image of the image pair and a feature map corresponding to a second image of the image pair.
26 . The method of claim 23 , further comprising determining a semantic correspondence between an object in a first image of the image pair and a target object in a second image of the image pair based on the correspondence result.