Image processing apparatus and image processing method
This invention improves the image quality of areas that the user want sharply focused. Depth values representing positions in a depth direction of scenes including the photographed object are estimated to generate a depth map. The focus map generation unit, based on the virtual focus parameter 305 and the depth map, generates by an area dividing operation a focus map representing an area to be sharply focused (in-focus area) and an area to be blurred (out-of-focus area). The in-focus area image processing unit places greater importance on the in-focus area in the image processing to generate an in-focus area image. The out-of-focus area image processing unit puts greater importance on the out-of-focus area in the image processing to generate an out-of-focus area image. The image blending unit generates a refocused image by blending the in-focus area image and the out-of-focus area image.
1. An image processing apparatus comprising:
a photographed data input unit configured to acquire a plurality of images photographed from at least two different viewing positions;
an input unit configured to input a refocus parameter representing a distance from a camera to a virtual focal plane;
a distance estimation unit configured to generate a depth map for the acquired images indicating distances between photographed objects corresponding to pixels and a camera;
a focus map generation unit configured to generate a focus map indicating an in-focus area to be sharply focused and an out-of-focus area to be blurred, by calculating areas including the in-focus area and the out-of-focus area from the depth map according to the refocus parameters;
an in-focus area image processing unit configured to generate a synthesized image by performing a resolution enhancement process on the in-focus area and synthesizing a group of images including at least the photographed images;
an out-of-focus area image processing unit configured to generate a synthesized image by performing an operation, different from the resolution enhancement process performed by the in-focus area image processing unit, on the out-of-focus area and synthesizing a group of images including at least the photographed images, wherein the out-of-focus area image processing unit further comprises:
an image reduction unit configured to reduce resolution of images in the out-of-focus area; and
an image interpolation unit to generate interpolated images by interpolating between the reduced images through interpolation between the viewing positions,
wherein the out-of-focus area image processing unit generates a synthesized image of the out-of-focus area by synthesizing the reduced images and the interpolated images,
wherein the out-of-focus area image processing unit generates a synthesized image at a resolution lower than that of the photographed images in the out-of-focus area using the image reduction unit, and
wherein the out-of-focus area imaging processing unit further includes an image enlargement unit configured to transform the synthesized image to the one with an output resolution; and
an image blending unit configured to generate a refocused image by blending the synthesized image of the in-focus area and the synthesized image of the out-of-focus area according to the focus map.
2. An image processing apparatus according to claim 1 , wherein a plurality of synthesized images with different resolutions are generated and held in the out-of-focus area.
3. An image processing method comprising:
a photographed data input step of acquiring a plurality of images photographed from at least two different viewing positions;
an input step of inputting a refocus parameter representing a distance from a camera to a virtual focal plane;
a distance estimation step of generating a depth map for the acquired images indicating distances between photographed objects corresponding to pixels and a camera;
a focus map generation step of generating a focus map indicating an in-focus area to be sharply focused and an out-of-focus area to be blurred, by calculating areas including the in-focus area and the out-of-focus area from the depth map according to the refocus parameters;
an in-focus area image processing step of generating a synthesized image by performing a resolution enhancement process on the in-focus area and synthesizing a group of images including at least the photographed images;
an out-of-focus area image processing step of generating a synthesized image by performing an operation, different from the resolution enhancement process performed by the in-focus area image processing step, on the out-of-focus area and synthesizing a group of images including at least the photographed images, wherein the out-of-focus area image processing step further comprises:
an image reduction step of reducing resolution of images in the out-of-focus area; and
an image interpolation step of generating interpolated images by interpolating between the reduced images through interpolation between the viewing positions,
wherein the out-of-focus area image processing step generates a synthesized image of the out-of-focus area by synthesizing the reduced images and the interpolated images,
wherein the out-of-focus area image processing unit generates a synthesized image at a resolution lower than that of the photographed images in the out-of-focus area using the image reduction unit;
wherein the out-of-focus area imaging processing unit further includes an image enlargement unit configured to transform the synthesized image to the one with an output resolution; and
an image blending step of generating a refocused image by blending the synthesized image of the in-focus area and the synthesized image of the out-of-focus area according to the focus map.