Image processing device, image processing method and image processing program

1. An image processing device for generating an output image as a color image wherein each pixel has color information, using a first image as a sparsely color sampled image wherein each pixel has monochromatic information, which is obtained by an image sensor which has multiple pixels that photoelectrically convert light of a plurality of colors, the image processing device comprising:

a simulation unit configured to generate a second image as an estimated image of the sparsely color sampled image by simulating imaging using the image sensor based on the first image as the color image;

a comparison unit configured to compare the second image with the first image, and to generate first information which shows a comparison result;

a converting unit configured to convert the first information into second information which corresponds to the color image;

a penalty computing unit configured to compute a correction amount of the first image based on an image penalty which is determined corresponding to the first image; and

a correction unit configured to generate a third image as the color image, by correcting the first image based on at least one of the second information and the correction amount;

wherein the third image which is generated by repeating the processing of the correction unit as the input image of the third image, becomes the output image.

2. The image processing device according to claim 1 , wherein the simulation unit is configured to generate the second image as the estimated image of the sparsely color sampled image by simulating imaging wherein the image sensor is used, including non-linear conversion, based on the first image of the color image.

3. The image processing device according to claim 1 , wherein the first image is acquired through a color filter, the simulation unit is configured to perform a color response conversion wherein the characteristics of the color filter are simulated, and the converting unit is configured to perform color distribution corresponding to the color response conversion.

4. The image processing device according to claim 1 , wherein the first image is acquired through an optical low-pass filter, the simulation unit is configured to perform convolution wherein the characteristics of the optical low-pass filter are simulated, and the converting unit is configured to perform inversion convolution as to the results of the convolution.

5. The image processing device according to claim 1 , wherein the comparison unit is configured to generate the first information illustrating a difference between the third image and the first image.

6. The image processing device according to claim 1 , wherein the comparison unit is configured to generate the first information illustrating a ratio between the third image and the first image.

7. The image processing device according to claim 1 , wherein the simulation unit is configured to perform linear conversion and nonlinear conversion which acts upon elements within a same basis space and does not reverse a magnitude correlation of the elements.

8. The image processing device according to claim 1 , wherein the correcting amount is a value which decreases the image penalty by adding to the first image.

9. The image processing device according to claim 1 , wherein the image penalty is expressed as a sum of pixel penalties capable of calculating by pixel.

10. The image processing device according to claim 1 , wherein the image penalty is expressed using a color plane penalty capable of calculating by color plane.

11. The image processing device according to claim 10 , wherein the color plane penalty reflects a correlation between the color plane and other color planes.

12. An imaging device comprising:

an image sensor; and

the image processing device according to claim 1 .

13. The imaging device according to claim 12 , wherein the image sensor is one of an image sensor wherein color filters of four or more colors are arranged, an image sensor wherein color filters of multiple colors are arranged in a cycle greater than 2×2, or an image sensor wherein the color filters of multiple colors are in an irregular arrangement.

14. An image processing device for generating an output image as a color image wherein each pixel has color information, using a first image as a sparsely color sampled image wherein each pixel has monochromatic information, which is obtained by an image sensor which has multiple pixels that photoelectrically convert light of a plurality of colors, the image processing device comprising:

a simulation unit configured to generate a second image as an estimated image of the sparsely color sampled image, by simulating imaging using the image sensor including nonlinear conversion based on the first image as the color image;

a comparison unit configured to compare the second image with the first image, and to generate first information which shows a comparison result;

a converting unit configured to convert the first image into the second information which corresponds to the color image; and

a correction unit configured to generate a third image as the color image, by correcting the input image based on the second information;

wherein the third image which is generated by repeating the processing of the correction unit as the input image of the third image, becomes the output image.

15. The image processing device according to claim 14 , wherein the nonlinear conversion is a nonlinear conversion which acts upon each element within a same basis space and does not reverse a magnitude correlation of the elements.

16. An imaging device comprising:

an image sensor; and

the image processing device according to claim 14 .

17. An image processing method for generating an output image as a color image wherein each pixel has color information, using a first image as a sparsely color sampled image wherein each pixel has monochromatic information, which is obtained by an image sensor which has multiple pixels that photoelectrically convert light of a plurality of colors, the method comprising:

a simulation step for generating a second image as an estimated image of the sparsely color sampled image by simulating imaging using the image sensor based on the first image as the color image;

a comparison step for comparing the second image with the first image, and generating first information which shows a comparison result;

a converting step for converting the first information into the second information which corresponds to the color image;

a penalty computing step for computing a correction amount of the input image based on the image penalty which is determined corresponding to the input image; and

a correction step for generating a third image as the color image, by correcting the input image based on at least one of the second information and the correcting amount;

wherein the third image which is generated by repeating the processing of the correction step as the input image of the third image, becomes the output image.

18. An image processing program for causing a computer to execute the image processing method according to claim 17 .

19. An image processing method for generating an output image as a color image wherein each pixel has color information, using a first image as a sparsely color sampled image wherein each pixel has monochromatic information, which is obtained by an image sensor which has multiple pixels that photoelectrically convert light of a plurality of colors, the method comprising:

a simulation step for generating a second image as an estimated image of the sparsely color sampled image by simulating imaging using the image sensor including nonlinear conversion based on the first image as the color image;

a comparison step for comparing the second image with the first image and generating first information which shows a comparison result;

a converting step for converting the first information into the second information which corresponds to the color image; and

a correction step for generating a third image as the color image by correcting the input image based on the second information;

wherein the third image which is generated by repeating the processing of the correction step as the input image of the third image, becomes the output image.

20. An image processing program for causing a computer to execute the image processing method according to claim 19 .