Image capturing apparatus, image capturing system, and image capturing method
An image capturing apparatus includes: an optical system that gives aberration to incident light; an image capturing unit that converts the light that has passed through the optical system into pixels, and captures an image; and an inverse transform unit that obtains a first inverse transform filter for restoring the aberration for each predetermined part of the captured image captured by the image capturing unit, and performs a first inverse transformation process on the captured image by the first inverse transform filter.
1. An image capturing apparatus comprising:
an optical system that gives aberration to incident light;
an image capturing sensor configured to convert the light that has passed through the optical system into pixels, and capture an image including the pixels; and
processing circuitry configured to perform a filtering process on the captured image based on a filter applicable to each predetermined portion of the captured image for compensating the aberration caused by the optical system, wherein the processing circuitry is configured to:
transform the captured image to a set of values in a frequency domain;
calculate a frequency characteristic of the filter in each predetermined portion using a value E[|S′(ω) | 2 ] of a local power spectrum of an ideal image corresponding to a frequency characteristic S′(ω) of a local part of the ideal image, the frequency characteristics S′(ω) being obtained by transforming pixels in the local part of the ideal image into the frequency domain, the local part corresponding to the respective predetermined portion and being smaller than the entire captured image;
generate a set of processed values by processing the set of values in the frequency domain based on the frequency characteristic of the filter; and
transform the set of processed values in the frequency domain to a filtered pixel corresponding to a pixel in the respective predetermined portion of the captured image,
wherein the processing circuitry is configured so that the frequency characteristic of the filter is a product of a frequency characteristic of a first filter for compensating the entire captured image for the aberration, and a frequency characteristic of a second filter for correcting the frequency characteristic of the first filter within the respective predetermined portion of the captured image.
2. The image capturing apparatus according to claim 1 , wherein the processing circuitry is configured so that the filtering process minimizes a mean square error between a portion of the ideal image of a subject to which the respective predetermined portion of the captured image corresponds and a corresponding portion of an output image on which the filtering process has been performed.
3. The image capturing apparatus according to claim 1 , wherein the processing circuitry is configured so that a frequency characteristic R′ of the filter is calculated by a following equation:
R
′
(
ω
)
=
H
(
ω
)
*
·
E
[
S
(
ω
)
2
]
H
(
ω
)
2
·
E
[
S
′
(
ω
)
2
]
+
E
[
W
(
ω
)
2
]
(R′: the frequency characteristic of the filter,
S′: the frequency characteristic of the local part of the ideal image,
W: a frequency characteristic of a noise,
H: a frequency characteristic of a point spread function of the optical system, and
ω: spatial frequency).
4. The image capturing apparatus according to claim 1 , wherein the processing circuitry is configured so that the frequency characteristic K of the second filter is calculated by a following equation:
K
(
ω
)
==
H
(
ω
)
2
E
[
W
(
ω
)
2
]
+
E
[
1
S
(
ω
)
2
]
H
(
ω
)
2
E
[
W
(
ω
)
2
]
+
E
[
1
S
′
(
ω
)
2
]
(K: the frequency characteristic of the second filter,
S: a frequency characteristic of an entire ideal image,
S′: the frequency characteristic of the local part of the ideal image,
W: a frequency characteristic of a noise,
H: a frequency characteristic of a point spread function of the optical system, and
ω: spatial frequency).
5. The image capturing apparatus according to claim 1 , wherein the processing circuitry is configured so that the frequency characteristic K of the second filter is calculated by a following equation:
K
(
ω
)
=
E
[
S
′
(
ω
)
2
]
E
[
S
(
ω
)
2
]
(K: the frequency characteristic of the second filter,
S: a frequency characteristic of an entire ideal image,
S′: the frequency characteristic of the local part of the ideal image, and
ω: spatial frequency).
6. The image capturing apparatus according to claim 1 , wherein the processing circuitry is configured so that the frequency characteristic K of the second filter is calculated by a following equation:
K (ω)= t·E[|S′ (ω)| 2 ]
(K: the frequency characteristic of the second filter,
t: a constant of proportionality,
S′: the frequency characteristic of the local part of the ideal image, and
ω: spatial frequency).
7. The image capturing apparatus according to claim 1 , wherein the processing circuitry is configured so that the frequency characteristic of the local part of the ideal image is calculated based on processing a corresponding part of the captured image using the first filter.
8. The image capturing apparatus according to claim 1 , wherein the processing circuitry is configured so that the frequency characteristic of the second filter is derived from a look-up table in which the frequency characteristic of the second filter is associated with a corresponding part of the ideal image.
9. The image capturing apparatus according to claim 1 , wherein the processing circuitry is configured to:
transform the captured image to the set of values in the frequency domain using a discrete cosine transformation; and
transform the set of processed values in the frequency domain to the filtered pixel corresponding to the pixel in the respective predetermined portion of the captured image using an inverse discrete cosine transformation.
10. The image capturing apparatus according to claim 1 , wherein the processing circuitry is configured so that a number of taps of the second filter is smaller than a number of taps of the first filter.
11. The image capturing apparatus according to claim 1 , wherein the processing circuitry is configured to generate the set of processed values in the frequency domain by computation of values in a real number space.
12. An image capturing system comprising:
an image capturing apparatus including:
an optical system that causes aberration to incident light;
an image capturing sensor configured to convert the light that has passed through the optical system into pixels, and capture an image including the pixels; and
processing circuitry configured to perform a filtering process on the captured image based on a filter applicable to each predetermined portion of the captured image for compensating the aberration caused by the optical system, wherein the processing circuitry is configured to:
transform the captured image to a set of values in a frequency domain;
calculate a frequency characteristic of the filter in each predetermined portion using a value E[|S′(ω)| 2 ] of a local power spectrum of an ideal image corresponding to a frequency characteristic S′(ω) of a local part of the ideal image, the frequency characteristics S′(ω) being obtained by transforming pixels in the local part of the ideal image into the frequency domain, the local part corresponding to the respective predetermined portion and being smaller than the entire captured image;
generate a set of processed values by processing the set of values in the frequency domain based on the frequency characteristic of the filter; and
transform the set of processed values in the frequency domain to a filtered pixel corresponding to a pixel in the respective predetermined portion of the captured image; and
an information processing apparatus including:
a communication sensor configured to receive, from the image capturing apparatus, an output image on which the filtering process has been performed, and
a display configured to display the output image,
wherein the processing circuitry of the image capturing apparatus is configured so that the frequency characteristic of the filter is a product of a frequency characteristic of a first filter for compensating the entire captured image for the aberration, and a frequency characteristic of a second filter for correcting the frequency characteristic of the first filter within the respective predetermined portion of the captured image.
13. An image capturing method comprising:
capturing an image with aberration caused by an optical system; and
performing a filtering process on the captured image based on a filter applicable to each predetermined portion of the captured image for compensating the aberration caused by the optical system, the filtering process including:
transforming the captured image to a set of values in a frequency domain;
calculating a frequency characteristic of the filter in each predetermined portion using a value E[|S′(ω)| 2 ] of a local power spectrum of an ideal image corresponding to a frequency characteristic S′(ω) of a local part of the ideal image, the frequency characteristics S′(ω) being obtained by transforming pixels in the local part of the ideal image into the frequency domain, the local part corresponding to the respective predetermined portion and being smaller than the entire captured image;
generating a set of processed values by processing the set of values in the frequency domain based on the frequency characteristic of the filter; and
transforming the set of processed values in the frequency domain to a filtered pixel corresponding to a pixel in the respective predetermined portion of the captured image,
wherein the frequency characteristic of the filter is a product of a frequency characteristic of a first filter for compensating the entire captured image for the aberration, and a frequency characteristic of a second filter for correcting the frequency characteristic of the first filter within the respective predetermined portion of the captured image.
14. The image capturing method according to claim 13 , wherein a frequency characteristic R′ of the filter is calculated by a following equation:
R
′
(
ω
)
H
(
ω
)
*
·
E
[
S
′
(
ω
)
2
]
H
(
ω
)
2
·
E
[
S
′
(
ω
)
2
]
+
E
[
W
(
ω
)
2
]
(R′: the frequency characteristic of the filter,
S′: the frequency characteristic of the local part of the ideal image,
W: a frequency characteristic of a noise,
H: a frequency characteristic of a point spread function of the optical system, and
ω: spatial frequency).
15. The image capturing method according to claim 13 , wherein the frequency characteristic K of the second filter is calculated by a following equation:
K
(
ω
)
==
H
(
ω
)
2
E
[
W
(
ω
)
2
]
+
E
[
1
S
(
ω
)
2
]
W
(
ω
)
2
E
[
W
(
ω
)
2
]
+
E
[
1
S
′
(
ω
)
2
]
(K: the frequency characteristic of the second filter,
S: a frequency characteristic of an entire ideal image,
S′: the frequency characteristic of the local part of the ideal image,
W: a frequency characteristic of a noise,
H: a frequency characteristic of a point spread function of the optical system, and
ω: spatial frequency).
16. The image capturing method according to claim 13 , wherein the frequency characteristic K of the second filter is calculated by a following equation:
K
(
ω
)
=
E
[
S
′
(
ω
)
2
]
E
[
S
(
ω
)
2
]
(K: the frequency characteristic of the second filter,
S: a frequency characteristic of an entire ideal image,
S′: the frequency characteristic of the local part of the ideal image, and
ω: spatial frequency).
17. The image capturing method according to claim 13 , wherein the frequency characteristic K of the second filter is calculated by a following equation:
K (ω)= t·E[|S′ (ω)| 2 ]
(K: the frequency characteristic of the second filter,
t: a constant of proportionality,
S′: the frequency characteristic of the local part of the ideal image, and
ω: spatial frequency).
18. The image capturing method according to claim 13 , wherein a number of taps of the second filter is smaller than a number of taps of the first filter.
19. The image capturing apparatus according to claim 1 , wherein the processing circuitry is configured so that the second filter is calculated using the value E[|S′(ω)| 2 ] of the local power spectrum that corresponds to the respective predetermined portion of the captured image.