Image providing device, image providing method, and image providing program for providing past-experience images

1. An image providing device for compositing foreground image data with background image data after applying the background image data to an inner surface of a three-dimensional model in a three-dimensional modeling space, the image providing device comprising:

a reader operable to access a recording medium to read a stereoscopic photograph file containing left-view photograph data, right-view photograph data, and geographic information;

a transmitter operable to generate, with use of the geographic information contained in the stereoscopic photograph file, an image acquisition request that specifies a location of a current viewpoint in the geographic information, and to transmit the image acquisition request to an image collection server;

a receiver operable to receive a street view file transmitted from the image collection server in response to the image acquisition request, the street view file containing the background image data and having a location attribute matching the geographic information included in the image acquisition request;

an extractor operable to extract feature points from foreground image data, which is one of the left-view photograph data and the right-view photograph data determined as base image data, and from the background image data;

a matcher operable to search for a pair of feature points matching between the foreground image data and the background image data and obtain a transformation matrix for projecting the foreground image data onto the background image data, by using the pair of matching feature points;

a transformer operable to determine the one of the left-view photograph data and the right-view photograph data as the base image data, by comparing an image capturing direction of the background data with an image capturing direction of each photographic image, and to apply, to foreground depth data associated with the foreground image data, a transformation represented by the transformation matrix obtained by the matcher;

a depth-based renderer operable to perform depth-based rendering of the foreground image data based on the transformed foreground depth data, thereby to generate two or more pieces of viewpoint image data corresponding to the foreground image data; and

a compositor operable to composite the two or more pieces of viewpoint image data corresponding to the foreground image data with two or more pieces of viewpoint image data corresponding to the background image data.

2. The image providing device according to claim 1 , wherein

a transformation matrix H is given by Formula 1 below,

Q

→

=

[

X

F

Y

F

Z

F

]

⁢

⁢

q

→

=

[

x

f

y

f

1

]

⁢

⁢

q

→

=

H

→

⁢

⁢

Q

→

⁢

⁢

where

[

Formula

⁢

⁢

1

]

xf and yf are X and Y coordinates of a feature point f(xf, yf) on the foreground image data,

XF and YF are X and Y coordinates of a corresponding feature point F(XF, YF) on the background image data to the feature point f(xf, yf), and

ZF is a depth value of the feature point F(XF, YF).

3. The image providing device according to claim 1 , further comprising:

an acquirer operable to acquire photographic image data captured by a stereoscopic camera, the captured photographic image data including the right-view photograph data and the left-view photograph data.

4. The image providing device according to claim 3 , further comprising:

a depth image generator operable to generate a depth image of the photographic image data from the right-view photograph data and the left-view photograph data, wherein

the foreground depth data subjected to the transformation by the transformer is the depth image generated by the depth image generator.

5. The image providing device according to claim 3 , wherein

one of the right-view photograph data and the left-view photograph data not determined as the base image data is designated as non-base image data,

the depth-based renderer is operable to re-generate image data by applying horizontal shift to pixels of the non-base image data according to depths represented by the foreground depth data, and

the two or more pieces of viewpoint image data are composed of the non-base image data and the re-generated image data.

6. The image providing device according to claim 1 , wherein

a permissible level of a depth of feature points to be searched for is set in advance in the image providing device, and

when searching the foreground image data for feature points for matching with a street view viewpoint, the matcher is operable to exclude any feature point having a median of depth distribution falling outside a permissible range, and perform matching between the background image data and the foreground image data using feature points each having a median of depth distribution falling within the permissible range.

7. The image providing device according to claim 1 , wherein

the background image is a panoramic image as seen from a geographic location corresponding to the geographic information included in the image acquisition request.

8. An image providing method for compositing foreground image data with background image data after applying the background image data to an inner surface of a three-dimensional model in a three-dimensional modeling space, the image providing method comprising:

accessing a recording medium to read a stereoscopic photograph file containing left-view photograph data, right-view photograph data, and geographic information;

generating, with use of the geographic information contained in the stereoscopic photograph file, an image acquisition request that specifies a location of a current viewpoint in the geographic information, and transmitting the image acquisition request to an image collection server;

receiving a street view file transmitted from the image collection server in response to the image acquisition request, the street view file containing the background image data and having a location attribute matching the geographic information included in the image acquisition request;

extracting feature points from the foreground image data, which is one of the left-view photograph data and the right-view photograph data determined as base image data, and from the background image data;

searching for a pair of feature points matching between the foreground image data and the background image data and obtaining a transformation matrix for projecting the foreground image data onto the background image data, by using the pair of matching feature points;

determining one of the left-view photograph data and the right-view photograph data as the base image data, by comparing an image capturing direction of the background image data with an image capturing direction of each photographic image, and applying, to foreground depth data associated with the foreground image data, a transformation represented by the transformation matrix obtained in the searching;

performing depth-based rendering of the foreground image data based on the transformed foreground depth data, thereby to generate two or more pieces of viewpoint image data corresponding to the foreground image data; and

compositing the two or more pieces of viewpoint image data corresponding to the foreground image data with two or more pieces of viewpoint image data corresponding to the background image data.

9. A non-transitory computer-readable medium including an image providing program for causing a computer to execute processing to composite foreground image data with background image data after applying the background image data to an inner surface of a three-dimensional model in a three-dimensional modeling space, the image providing program causing the computer to perform:

accessing a recording medium to read a stereoscopic photograph file containing left-view photograph data, right-view photograph data, and geographic information;

generating, with use of the geographic information contained in the stereoscopic photograph file, an image acquisition request that specifies a location of a current viewpoint in the geographic information, and transmitting the image acquisition request to an image collection server;

receiving a street view file transmitted from the image collection server in response to the image acquisition request, the street view file containing the background image data and having a location attribute matching the geographic information included in the image acquisition request;

extracting feature points from the foreground image data, which is one of the left-view photograph data and the right-view photograph data determined as base image data, and from the background image data;

searching for a pair of feature points matching between the foreground image data and the background image data and obtaining a transformation matrix for projecting the foreground image data onto the background image data, by using the pair of matching feature points;

determining one of the left-view photograph data and the right-view photograph data as the base image data, by comparing an image capturing direction of the background image data with an image capturing direction of each photographic image, and applying, to foreground depth data associated with the foreground image data, a transformation represented by the transformation matrix obtained in the searching;

performing depth-based rendering of the foreground image data based on the transformed foreground depth data, thereby to generate two or more pieces of viewpoint image data corresponding to the foreground image data; and

compositing the two or more pieces of viewpoint image data corresponding to the foreground image data with two or more pieces of viewpoint image data corresponding to the background image data.