Granted Patent B2

US 12,499,658 · App. 18/009,218 · Granted Dec 16, 2025

Inventors: Royston Rodrigues (Tokyo, JP); Masahiro Tani (Tokyo, JP)

Assignee: NEC CORPORATION

G06V10/774G06T3/02G06T5/77G06T7/11G06V10/44G06V10/761G06V10/7715G06V10/82G06V20/17G06T2207/10032G06T2207/20081G06T2207/20084G06T2207/20132

Abstract

Claims (84)

1 . A training apparatus comprising:

at least one processor; and

memory storing instructions,

wherein the at least one processor is configured to execute the instructions to:

perform a first phase training of a discriminator, the discriminator acquiring a ground-view image and an aerial-view image and determining whether the acquired ground-view image matches the acquired aerial-view image; and

perform a second phase training of the discriminator,

the first phase training is performed using the ground-view image, a positive example of the aerial-view image, and a first level negative example of the aerial-view image, the first level negative example of the aerial-view image including scenery of a different type from scenery in the ground-view image; and

the second phase training is performed using the ground-view image, a positive example of the aerial-view image, and a second level negative example of the aerial-view image, the second level negative example of the aerial-view image including scenery of a same type as scenery in the ground-view image,

wherein the at least one processor is further configured to execute the instructions to:

acquire a third negative example of the aerial-view image, whose degree of similarity to the ground-view image is higher than the degree of similarity of the second negative example of the aerial-view image to the ground-view image;

perform a third phase training of the discriminator using the ground-view image and the third negative example of the aerial-view image;

acquire a third negative example of the aerial-view image, whose degree of similarity to the ground-view image is higher than the degree of similarity of the second negative example of the aerial-view image to the ground-view image; and

perform a third phase training of the discriminator using the ground-view image and the third negative example of the aerial-view image.

2 . A training apparatus according to claim 1 ,

wherein the at least one processor is further configured to:

acquire a positive example of the aerial-view image and plural negative examples of the aerial-view images;

determine a type of scenery of each of the respective aerial-view images acquired;

use the negative example of the aerial-view image whose scenery is determined to be a different type from the scenery of the positive example of the aerial-view image, as the first level negative example of the aerial-view image; and

use the negative example of the aerial-view image whose scenery is determined to be a same type as the scenery of the positive example of the aerial-view image, as the second level negative example of the aerial-view image.

3 . The training apparatus according to claim 1 ,

wherein the transformation includes a crop operation, an affine transformation, color modification, or image inpainting,

the transformation parameters specify a part of the second level negative example of the aerial-view image to be cropped, in the case where the transformation includes a crop operation,

the transformation parameters include a transformation matrix with which an affine transformation on the second level negative example of the aerial-view image is performed, in the case where the transformation includes an affine transformation,

the transformation parameters include sets of location of pixel of the second level negative example of the aerial-view image and modification in color features to be applied to the corresponding pixel, in the case where the transformation includes a color modification, and

the transformation parameters include sets of location of pixel of the second level negative example of the aerial-view image and a new pixel value to be set to the corresponding pixel, in the case where the transformation includes an image inpainting.

4 . The training apparatus according to claim 1 ,

wherein the discriminator includes a first neural network, a second neural network, and a third neural network,

the first neural network acquires the ground-view image, and outputs feature maps of the ground-view image,

the second neural network acquires the aerial-view image, and outputs feature maps of the aerial-view image, and

the third neural network acquires the feature maps of the ground-view image and the feature maps of the aerial-view image, and outputs a data indicating whether the ground-view image matches the aerial-view image.

5 . A control method performed by a computer, comprising:

performing a first phase training of a discriminator, the discriminator acquiring a ground-view image and an aerial-view image and determining whether the acquired ground-view image matches the acquired aerial-view image; and

performing a second phase training of the discriminator,

wherein the first phase training is performed using the ground-view image, a positive example of the aerial-view image, and a first level negative example of the aerial-view image, the first level negative example of the aerial-view image including scenery of a different type from scenery in the ground-view image; and

the second phase training is performed using the ground-view image and a second level negative example of the aerial-view image, the second level negative example of the aerial-view image including scenery of a same type as scenery in the ground-view image,

the method further comprising:

acquiring a third negative example of the aerial-view image, whose degree of similarity to the ground-view image is higher than the degree of similarity of the second negative example of the aerial-view image to the ground-view image;

performing a third phase training of the discriminator using the ground-view image and the third negative example of the aerial-view image;

extracting features of the ground-view image and the second level negative example of the aerial-view image;

creating transformation parameters based on the extracted features of the ground-view image and the second level negative example of the aerial-view image; and

acquiring the third level negative example by transforming the second level negative example of the aerial-view image to the third level negative example of the aerial-view image based on the transformation parameters.

6 . The control method according to claim 5 , further comprising:

acquiring a positive example of the aerial-view image and plural negative examples of the aerial-view image;

determining a type of scenery of each of the respective aerial-view images acquired;

using the negative example of the aerial-view image whose scenery is determined to be a different type from the scenery of the positive example of the aerial-view image, as the first level negative example of the aerial-view image; and

using the negative example of the aerial-view image whose scenery is determined to be a same type as the scenery of the positive example of the aerial-view image, as the second level negative example of the aerial-view image.

7 . The control method according to claim 5 ,

wherein the transformation includes a crop operation, an affine transformation, color modification, or image inpainting,

the transformation parameters specify a part of the second level negative example of the aerial-view image to be cropped, in the case where the transformation includes a crop operation,

the transformation parameters include a transformation matrix with which an affine transformation on the second level negative example of the aerial-view image is performed, in the case where the transformation includes an affine transformation,

the transformation parameters include sets of location of pixel of the second level negative example of the aerial-view image and modification in color features to be applied to the corresponding pixel, in the case where the transformation includes a color modification, and

the transformation parameters include sets of location of pixel of the second level negative example of the aerial-view image and a new pixel value to be set to the corresponding pixel, in the case where the transformation includes an image inpainting.

8 . The control method according to claim 5 ,

wherein the discriminator includes a first neural network, a second neural network, and a third neural network,

the first neural network acquires the ground-view image, and outputs feature maps of the ground-view image,

the second neural network acquires the aerial-view image, and outputs feature maps of the aerial-view image, and

the third neural network acquires the feature maps of the ground-view image and the feature maps of the aerial-view image, and outputs a data indicating whether the ground-view image matches the aerial-view image.

9 . A non-transitory computer-readable storage medium storing a program that causes a computer to execute:

performing a first phase training of a discriminator, the discriminator acquiring a ground-view image and an aerial-view image and determining whether the acquired ground-view image matches the acquired aerial-view image; and

performing a second phase training of the discriminator,

wherein the first phase training is performed using the ground-view image, a positive example of the aerial-view image, and a first level negative example of the aerial-view image, the first level negative example of the aerial-view image including scenery of a different type from scenery in the ground-view image; and

the second phase training is performed using the ground-view image, a positive example of the aerial-view image, and a second level negative example of the aerial-view image, the second level negative example of the aerial-view image including scenery of a same type as scenery in the ground-view image,

the program further causes the computer to execute

acquiring a third negative example of the aerial-view image, whose degree of similarity to the ground-view image is higher than the degree of similarity of the second negative example of the aerial-view image to the ground-view image;

performing a third phase training of the discriminator using the ground-view image and the third negative example of the aerial-view image;

extracting features of the ground-view image and the second level negative example of the aerial-view image;

creating transformation parameters based on the extracted features of the ground-view image and the second level negative example of the aerial-view image; and

acquiring the third level negative example by transforming the second level negative example of the aerial-view image to the third level negative example of the aerial-view image based on the transformation parameters.

10 . The non-transitory computer-readable storage medium according to claim 9 , the program further causes the computer to execute:

acquiring a positive example of the aerial-view image and plural negative examples of the aerial-view images;

determining a type of scenery of each of the respective aerial-view images acquired;

using the negative example of the aerial-view image whose scenery is determined to be a different type from the scenery of the positive example of the aerial-view image, as the first level negative example of the aerial-view image; and

using the negative example of the aerial-view image whose scenery is determined to be a same type as the scenery of the positive example of the aerial-view image, as the second level negative example of the aerial-view image.

11 . The non-transitory computer-readable storage medium according to claim 9 ,

wherein the transformation includes a crop operation, an affine transformation, color modification, or image inpainting,

the transformation parameters specify a part of the second level negative example of the aerial-view image to be cropped, in the case where the transformation includes a crop operation,

the transformation parameters include a transformation matrix with which an affine transformation on the second level negative example of the aerial-view image is performed, in the case where the transformation includes an affine transformation,

the transformation parameters include sets of location of pixel and modification in color features to be applied to the corresponding pixel, and

the transformation parameters include sets of location of pixel and a new pixel value to be set to the corresponding pixel.

12 . The non-transitory computer-readable storage medium according to claim 9 ,

wherein the discriminator includes a first neural network, a second neural network, and a third neural network,

the first neural network acquires the ground-view image, and outputs feature maps of the ground-view image,

the second neural network acquires the aerial-view image, and outputs feature maps of the aerial-view image, and

the third neural network acquires the feature maps of the ground-view image and the feature maps of the aerial-view image, and outputs a data indicating whether the ground-view image matches the aerial-view image.

Assignments (1)

Continuity (1)

References Cited (9)