Image generation apparatus and image generation method

1. An image generation apparatus for generating, based on an input image and a depth map corresponding to the input image, an image of a new viewpoint of a scene represented in the input image, the apparatus comprising:

a processor;

a pixel-shifted image generation unit configured to generate, using the processor, a first pixel-shifted image and a second pixel-shifted image, the first pixel-shifted image resulting from a process of shifting, based on the input image and the depth map, a pixel in the input image by a shift amount corresponding to the new viewpoint, the second pixel-shifted image resulting from a process of shifting a pixel in a background edge image by the same shift amount as for the first pixel-shifted image, and the background edge image being included in the input image and containing a designated edge pixel that is a pixel located in a background region around an edge of a foreground object;

an image fusion unit configured to fuse the second pixel-shifted image and the first pixel-shifted image to include a background region in the second pixel-shifted image into a hole that is a region having no pixel information and created in the first pixel-shifted image in the process of shifting the pixel; and

a hole filling unit configured to fill the hole with pixel information based on the fused pixel-shifted image.

2. The image generation apparatus according to claim 1 , further comprising

a background edge image generation unit including:

an edge detection module which detects an edge included in the input image and an edge included in the depth map;

an edge filtering module which aligns a first edge image and a second edge image, the first edge image indicating edge information included in the input image, and the second edge image indicating edge information included in the depth map; and

a generation module which generates the background edge image by replacing, based on the edge included in the input image after the aligning and the edge included in the depth map after the aligning, a pixel value of the edge included in the input image with a pixel value of a pixel indicating a maximum depth position within a predetermined range from, on the depth map, a position corresponding to a position of the edge included in the input image.

3. The image generation apparatus according to claim 2 ,

wherein the edge filtering module determines a first edge pixel and a second edge pixel, and aligns the first edge image and the second edge image by matching a position of one of the first edge pixel and the second edge pixel to a position of the other, the first edge pixel indicating the edge within a predetermined region in one of the first edge image and the second edge image, and the second pixel being closest in distance to the first edge pixel among pixels indicating the edge within the predetermined region in the other of the first edge image and the second edge image.

4. The image generation apparatus according to claim 2 ,

wherein, in each of the first edge image and the second edge image, the edge has two or more pixels in a width direction.

5. The image generation apparatus according to claim 4 ,

wherein, among the pixels of the edge in the width direction, at least one pixel is included in the foreground object, and at least one other pixel is included in the background region.

6. The image generation apparatus according to claim 2 ,

wherein the edge detection module includes an edge detector included in a group of a Canny edge detector, a Prewitt edge detector, a Sobel edge detector, a Laplacian of Gaussian edge detector, and other differential edge detectors.

7. The image generation apparatus according to claim 1 ,

wherein the pixel-shifted image generation unit is configured to calculate the shift amount based on the depth map and at least one parameter indicating a gain of the shift amount.

8. The image generation apparatus according to claim 1 ,

wherein the image fusion unit is configured to fuse the second pixel-shifted image onto the first pixel-shifted image by replacing a pixel value of the first pixel-shifted image with a corresponding pixel value of the second pixel-shifted image.

9. The image generation apparatus according to claim 1 ,

wherein the image fusion unit is configured to fuse the second pixel-shifted image onto the first pixel-shifted image based on a weighted average of pixel values.

10. The image generation apparatus according to claim 1 ,

wherein the hole filling unit is configured to fill the hole with the pixel information by performing a total variation (TV) inpainting process on the fused pixel-shifted image.

11. The image generation apparatus according to claim 10 ,

wherein the hole filling unit is configured to fill the hole with the pixel information in the TV inpainting process based on a change rate of pixel values which is computed using neighborhood pixels.

12. The image generation apparatus according to claim 1 ,

wherein the hole filling unit is further configured to perform, on the fused pixel-shifted image which has the hole filled by the hole filling unit, a process of smoothing a boundary of the foreground object based on a similarity of pixel values around the boundary and a depth degree indicated by the depth map.

13. An image generation method for generating, based on an input image and a depth map corresponding to the input image, an image of a new viewpoint of a scene represented in the input image, the method comprising:

generating, using a processor, a first pixel-shifted image and a second pixel-shifted image, the first pixel-shifted image resulting from a process of shifting, based on the input image and the depth map, a pixel in the input image by a shift amount corresponding to the new viewpoint, the second pixel-shifted image resulting from a process of shifting a pixel in a background edge image by the same shift amount as for the first pixel-shifted image, and the background edge image being included in the input image and containing a designated edge pixel that is a pixel located in a background region around an edge of a foreground object;

fusing the second pixel-shifted image and the first pixel-shifted image to include a background region in the second pixel-shifted image into a hole that is a region having no pixel information and created in the first pixel-shifted image in the process of shifting the pixel; and

filling the hole with pixel information based on the fused pixel-shifted image.

14. A non-transitory computer-readable recording medium having recorded thereon a program for causing a computer to execute the image generation method according to claim 13 .

15. An integrated circuit executing the image generation method according to claim 13 .