Image processing apparatus, endoscope, and computer readable medium
An image processing apparatus includes a color image data generation section 54 that generates color image data from an image pickup signal of a red component, an image pickup signal of a green component, and an image pickup signal of a blue component which are output from an image pickup device 100 ; an infrared image data generation section 53 that generates infrared image data from an image pickup signal of an infrared component output from the image pickup device 100 ; and a high-contrast infrared image data generation section 55 that generates high-contrast infrared image data using the color image data and the infrared image data. Contrast of the high-contrast infrared image data is more enhanced than that of the infrared image data.
1. An image processing apparatus for generating image data from an image pickup signal output from an image pickup device, the apparatus comprising:
a color image data generation unit that generates color image data from an image pickup signal of a red component, an image pickup signal of a green component, and an image pickup signal of a blue component which are output from the image pickup device;
an infrared image data generation unit that generates infrared image data from an image pickup signal of an infrared component output from the image pickup device; and
a high-contrast infrared image data generation unit that generates high-contrast infrared image data using the color image data and the infrared image data, wherein: contrast of the high-contrast infrared image data is more enhanced than that of the infrared image data,
wherein:
pixel data of the color image data includes red-component data, green-component data and blue-component data,
pixel data of the infrared image data includes infrared-component data, the high-contrast infrared image data generation unit generates the high-contrast infrared image data using the following expression:
I ( x,y )= r 1 ×R ( x,y )+ g 1 ×G ( x,y )+ b 1 ×B ( x,y )+ ir 1 ×IR ( x,y )
where I(x, y) denotes pixel data of the high-contrast infrared image data at coordinates (x, y),
R(x, y) denotes the red-component data of the pixel data at the coordinates (x, y),
G(x, y) denotes the green-component data of the pixel data at the coordinates (x, y),
B(x, y) denotes the blue-component data of the pixel data at the coordinates (x, y), and
r 1 , g 1 , b 1 and ir 1 denote coefficients, and
the coefficients r 1 , g 1 , b 1 , and ir 1 are determined so that r 1 ×R(λ)+g 1 ×G(λ)+b 1 ×B(λ)+ir 1 ×IR(λ) is as close as possible to Real(λ), where
R(λ) denotes a spectral sensitivity of a red-component photoelectric conversion element of the photoelectric conversion elements that outputs the image pickup signal of the red component,
G(λ) denotes a spectral sensitivity of a green-component photoelectric conversion element of the photoelectric conversion elements that outputs the image pickup signal of the green component,
B(λ) denotes a spectral sensitivity of a blue-component photoelectric conversion element of the photoelectric conversion elements that outputs the image pickup signal of the blue component,
IR(λ) denotes a spectral sensitivity of an infrared-component photoelectric conversion element of the photoelectric conversion elements that outputs the image pickup signal of the infrared component, and
Real (λ) denotes a spectral sensitivity with which a observation target substance used to grasp change in a human body can be picked up with the highest contrast.
2. The apparatus according to claim 1 , wherein the observation target substance is hemoglobin.
3. The apparatus according to claim 1 , further comprising:
a high-color-reproduction color image data generation unit that generates high-color-reproduction color image data using the color image data and the infrared image data, wherein:
the high-color-reproduction color image data has color reproducibility higher than the color image data.
4. The apparatus according to claim 3 , wherein:
the high-color-reproduction color image data generation unit generates each pixel data of the high-color-reproduction color image data using the following expression:
(
R
O
(
x
,
y
)
G
O
(
x
,
y
)
B
O
(
x
,
y
)
)
=
(
r
2
g
2
b
2
i
r
2
r
3
g
3
b
3
i
r
3
r
4
g
4
b
4
i
r
4
)
(
R
(
x
,
y
)
G
(
x
,
y
)
B
(
x
,
y
)
I
r
(
x
,
y
)
)
where
R O (x, y) denotes red-component data of the pixel data of the high-color-reproduction color image data at coordinates (x, y),
G O (x, y) denotes green-component data of the pixel data of the high-color-reproduction color image data at the coordinates (x, y),
B O (x, y) denotes blue-component data of the pixel data of the high-color-reproduction color image data at the coordinates (x, y),
R(x, y) denotes the red-component data of the pixel data of the color image data at the coordinates (x, y),
G(x, y) denotes the green-component data of the pixel data of the color image data at the coordinates (x, y),
B(x, y) denotes the blue-component data of the pixel data of the color image data at the coordinates (x, y),
Ir(x, y) denotes the infrared-component data of the pixel data of the color image data at the coordinates (x, y), and
r 2 , r 3 , r 4 , g 2 , g 3 , g 4 , b 2 , b 3 , b 4 , ir 2 , ir 3 and ir 4 denote coefficients,
the coefficients r 2 , g 2 , b 2 , and ir 2 are determined so that r 2 ×R(λ)+g 2 ×G(λ)+b 2 ×B(λ)+ir 2 ×IR(λ) is as close as possible to R O (λ), where
R(λ) denotes a spectral sensitivity of a red-component photoelectric conversion element of the photoelectric conversion elements that outputs the image pickup signal of the red component,
G(λ) denotes a spectral sensitivity of a green-component photoelectric conversion element of the photoelectric conversion elements that outputs the image pickup signal of the green component,
B(λ) denotes a spectral sensitivity of a blue-component photoelectric conversion element of the photoelectric conversion elements that outputs the image pickup signal of the blue component,
IR(λ) denotes a spectral sensitivity of an infrared-component photoelectric conversion element of the photoelectric conversion elements that outputs the image pickup signal of the infrared component, and
R O (λ) denotes an ideal spectral sensitivity of a photoelectric conversion element that outputs an image pickup signal of the red component,
the coefficients r 3 , g 3 , b 3 , and ir 3 are determined so that r 3 ×R(λ)+g 3 ×G(λ)+b 3 ×B(λ)+ir 3 ×IR(λ) is as close as possible to G O (λ), where G O (λ) denotes an ideal spectral sensitivity of a photoelectric conversion element that outputs an image pickup signal of the green component, and
the coefficients r 4 , g 4 , b 4 , and ir 4 are determined so that r 4 ×R(λ)+g 4 ×G(λ)+b 4 ×B(λ)+ir 4 ×IR(λ) is as close as possible to B O (λ), where G O (λ) denotes an ideal spectral sensitivity of a photoelectric conversion element that outputs an image pickup signal of the blue component.
5. The apparatus according to claim 1 , wherein the image pickup device comprises:
a large number of first photoelectric conversion elements arranged on a first plane in a semiconductor substrate;
second photoelectric conversion elements formed on a second plane, which is located above the first photoelectric conversion elements, the second photoelectric conversion elements that correspond to a part of the large number of first photoelectric conversion elements, the second photoelectric conversion elements comprising:
first electrodes formed above the first photoelectric conversion elements,
a photoelectric conversion layer formed on the first electrodes, and
a second electrode formed on the photoelectric conversion layer;
a color filter layer formed above the first photoelectric conversion elements, the color filter layer that transmits light in a wavelength range different from a wavelength range of light that the photoelectric conversion layer absorbs; and
a signal reading portion that reads (i) signals that correspond to charges generated in the second photoelectric conversion elements and (ii) signals that correspond to charges generated in the first photoelectric conversion elements, wherein:
the color filter layer comprises a large number of color filters that correspond to the large number of photoelectric conversion elements, respectively,
the large number of color filters are classified into three types of color filters of those for transmitting light in a red wavelength range, those for transmitting light in a green wavelength range, and those for transmitting light in a blue wavelength range,
of the three types of color filters, at least the color filters for transmitting light in the red wavelength range also transmit infrared region light,
the photoelectric conversion layer absorbs the infrared region light to generate charges in response thereto, and transmits any other light than the infrared region light, and
the part of the large number of first photoelectric conversion elements are the first photoelectric conversion elements corresponding to the color filters for transmitting light in the red wavelength range.
6. The apparatus according to claim 5 , wherein the color filter layer is formed above the second photoelectric conversion elements.
7. The apparatus according to claim 6 , wherein:
the photoelectric conversion layer contains an organic material, and the image pickup device further comprises a protective layer that protects the second photoelectric conversion elements, the protective layer formed by an atomic layer chemical vapor deposition (ALCVD) method between the first photoelectric conversion elements and the color filter layer.
8. The apparatus according to claim 7 , wherein the protective layer contains an inorganic material.
9. The apparatus according to claim 8 , wherein the protective layer has a two-layer structure comprising an inorganic layer made of an inorganic material and an organic layer made of an organic polymer.
10. The apparatus according to claim 5 , wherein the image pickup device further comprises a microlens that collects light in each of the large number of first photoelectric conversion elements.
11. An endoscope comprising:
an image processing apparatus; and
the image processing apparatus according to claim 1 .
12. A non-transitory computer readable medium storing a program for causing a computer to execute a process for image processing, the image processing comprising:
generating color image data from an image pickup signal of a red component, an image pickup signal of a green component, and an image pickup signal of a blue component which are output from an image pickup device;
generating infrared image data from an image pickup signal of an infrared component output from the image pickup device; and
generating high-contrast infrared image data using the color image data and the infrared image data, wherein: contrast of the high-contrast infrared image data is more enhanced than that of the infrared image data,
wherein:
I ( x,y )= r 1 ×R ( x,y )+ g 1 ×G ( x,y )+ b 1 ×B ( x,y )+ ir 1 ×IR ( x,y )
where I(x, y) denotes pixel data of the high-contrast infrared image data at coordinates
R(x, y) denotes the red-component data of the pixel data at the coordinates (x, y),
G(x, y) denotes the green-component data of the pixel data at the coordinates (x, y),
B(x, y) denotes the blue-component data of the pixel data at the coordinates (x, y), and
r 1 , g 1 , b 1 and ir 1 denote coefficients, and
the coefficients r 1 , g 1 , b 1 , and ir 1 are determined so that r 1 ×R(λ)+g 1 ×G(λ)+b 1 ×B(λ)+ir 1 ×IR(λ) is as close as possible to Real (λ), where
R(λ) denotes a spectral sensitivity of a red-component photoelectric conversion element of the photoelectric conversion elements that outputs the image pickup signal of the red component,
G(λ) denotes a spectral sensitivity of a green-component photoelectric conversion element of the photoelectric conversion elements that outputs the image pickup signal of the green component,
B(λ) denotes a spectral sensitivity of a blue-component photoelectric conversion element of the photoelectric conversion elements that outputs the image pickup signal of the blue component,
IR(λ) denotes a spectral sensitivity of an infrared-component photoelectric conversion element of the photoelectric conversion elements that outputs the image pickup signal of the infrared component, and
Real (λ) denotes a spectral sensitivity with which a observation target substance used to grasp change in a human body can be picked up with the highest contrast.
13. An image processing apparatus for generating image data from an image pickup signal output from an image pickup device, the apparatus comprising:
a color image data generation unit that generates color image data from an image pickup signal of a red component, an image pickup signal of a green component, and an image pickup signal of a blue component which are output from the image pickup device;
an infrared image data generation unit that generates infrared image data from an image pickup signal of an infrared component output from the image pickup device; and
a high-color-reproduction color image data generation unit that generates high-color-reproduction color image data using the color image data and the infrared image data, wherein:
the high-color-reproduction color image data has color reproducibility higher than the color image data,
wherein:
the high-color-reproduction color image data generation unit generates each pixel data of the high-color-reproduction color image data using the following expression:
(
R
o
(
x
,
y
)
G
o
(
x
,
y
)
B
o
(
x
,
y
)
)
=
(
r
2
g
2
b
2
ir
2
r
3
g
3
b
3
ir
3
r
4
g
4
b
4
ir
4
)
(
R
(
x
,
y
)
G
(
x
,
y
)
B
(
x
,
y
)
Ir
(
x
,
y
)
)
where Ro(x, y) denotes red-component data of the pixel data of the high-color-reproduction color image data at coordinates (x, y),
Go(x, y) denotes green-component data of the pixel data of the high-color-reproduction color image data at the coordinates (x, y),
Bo(x, y) denotes blue-component data of the pixel data of the high-color-reproduction color image data at the coordinates (x, y),
R(x, y) denotes the red-component data of the pixel data of the color image data at the coordinates (x, y),
G(x, y) denotes the green-component data of the pixel data of the color image data at the coordinates (x, y),
B(x, y) denotes the blue-component data of the pixel data of the color image data at the coordinates (x, y),
Ir(x, y) denotes the infrared-component data of the pixel data of the color image data at the coordinates (x, y), and
r 2 , r 3 , r 4 , g 2 , g 3 , g 4 , b 2 , b 3 , b 4 , ir 2 , ir 3 and ir 4 denote coefficients,
the coefficients r 2 , g 2 , b 2 , and ir 2 are determined so that r 2 ×R(λ)+g 2 ×G(λ)+b 2 ×B(λ)+ir 2 ×IR(λ)) is as close as possible to Ro(λ), where
R(λ) denotes a spectral sensitivity of a red-component photoelectric conversion element of the photoelectric conversion elements that outputs the image pickup signal of the red component,
G(λ) denotes a spectral sensitivity of a green-component photoelectric conversion element of the photoelectric conversion elements that outputs the image pickup signal of the green component,
B(λ) denotes a spectral sensitivity of a blue-component photoelectric conversion element of the photoelectric conversion elements that outputs the image pickup signal of the blue component,
IR(λ) denotes a spectral sensitivity of an infrared-component photoelectric conversion element of the photoelectric conversion elements that outputs the image pickup signal of the infrared component, and
Ro(λ) denotes an ideal spectral sensitivity of a photoelectric conversion element that outputs an image pickup signal of the red component
the coefficients r 3 , g 3 , b 3 , and ir 3 are determined so that r 3 ×R(λ)+g 3 ×G(λ)+b 3 ×B(λ)+ir 3 ×IR(λ) is as close as possible to Go(λ), where
Go(λ) denotes an ideal spectral sensitivity of a photoelectric conversion element that outputs an image pickup signal of the green component, and
the coefficients r 4 , g 4 , b 4 , and ir 4 are determined so that r 4 ×R(λ)+g 4 ×G(λ)+b 4 ×B(λ)+ir 4 ×IR(λ) is as close as possible to Bo(λ), where
Go(λ) denotes an ideal spectral sensitivity of a photoelectric conversion element that outputs an image pickup signal of the blue component.
14. The apparatus according to claim 13 , wherein the image pickup device comprises:
a large number of first photoelectric conversion elements arranged on a first plane in a semiconductor substrate;
second photoelectric conversion elements formed on a second plane, which is located above the first photoelectric conversion elements, the second photoelectric conversion elements that correspond to a part of the large number of first photoelectric conversion elements, the photoelectric conversion elements comprising:
first electrodes formed above the first photoelectric conversion elements,
a photoelectric conversion layer formed on the first electrodes, and
a second electrode formed on the photoelectric conversion layer;
a color filter layer formed above the first photoelectric conversion elements, the color filter layer that transmits light in a wavelength range different from a wavelength range of light that the photoelectric conversion layer absorbs; and
a signal reading portion that reads (i) signals that correspond to charges generated in the second photoelectric conversion elements and (ii) signals that correspond to charges generated in the first photoelectric conversion elements, wherein:
the color filter layer comprises a large number of color filters that correspond to the large number of photoelectric conversion elements, respectively,
the large number of color filters are classified into three types of color filters of those for transmitting light in a red wavelength range, those for transmitting light in a green wavelength range, and those for transmitting light in a blue wavelength range,
of the three types of color filters, at least the color filters for transmitting light in the red wavelength range also transmit infrared region light,
the photoelectric conversion layer absorbs the infrared region light to generate charges in response thereto, and transmits any other light than the infrared region light, and
the part of the large number of first photoelectric conversion elements are the first photoelectric conversion elements corresponding to the color filters for transmitting light in the red wavelength range.
15. The apparatus according to claim 14 , wherein the color filter layer is formed above the second photoelectric conversion elements.
16. The apparatus according to claim 15 , wherein:
the photoelectric conversion layer contains an organic material, and
the image pickup device further comprises a protective layer that protects the second photoelectric conversion elements, the protective layer formed by an atomic layer chemical vapor deposition (ALCVD) method between the first photoelectric conversion elements and the color filter layer.
17. The apparatus according to claim 16 , wherein the protective layer contains an inorganic material.
18. The apparatus according to claim 17 , wherein the protective layer has a two-layer structure comprising an inorganic layer made of an inorganic material and an organic layer made of an organic polymer.
19. The apparatus according to claim 14 , wherein the image pickup device further comprises a microlens that collects light in each of the large number of first photoelectric conversion elements.
20. An endoscope comprising:
an image processing apparatus; and
the image processing apparatus according to claim 13 .
21. A non-transitory computer readable medium storing a program for causing a computer to execute a process for image processing, the image processing comprising:
generating color image data from an image pickup signal of a red component, an image pickup signal of a green component, and an image pickup signal of a blue component which are output from an image pickup device;
generating infrared image data from an image pickup signal of an infrared component output from the image pickup device; and
generating high-color-reproduction color image data using the color image data and the infrared image data, wherein:
the high-color-reproduction color image data has color reproducibility higher than the color image data,
wherein:
I ( x,y )= r 1 ×R ( x,y )+ g 1 ×G ( x,y )+ b 1 ×B ( x,y )+ ir 1 ×IR ( x,y )
where I(x, y) denotes pixel data of the high-contrast infrared image data at coordinates (x, y),
R(x, y) denotes the red-component data of the pixel data at the coordinates (x, y),
G(x, y) denotes the green-component data of the pixel data at the coordinates (x, y),
B(x, y) denotes the blue-component data of the pixel data at the coordinates (x, y), and
r 1 , g 1 , b 1 and ir 1 denote coefficients, and
the coefficients r 1 , g 1 , b 1 , and ir 1 are determined so that r 1 ×R(λ)+g 1 ×G(λ)+b 1 ×B(λ)+ir 1 ×IR(λ) is as close as possible to Real (λ), where
R(λ) denotes a spectral sensitivity of a red-component photoelectric conversion element of the photoelectric conversion elements that outputs the image pickup signal of the red component,
G(λ) denotes a spectral sensitivity of a green-component photoelectric conversion element of the photoelectric conversion elements that outputs the image pickup signal of the green component,
B(λ) denotes a spectral sensitivity of a blue-component photoelectric conversion element of the photoelectric conversion elements that outputs the image pickup signal of the blue component,
IR(λ) denotes a spectral sensitivity of an infrared-component photoelectric conversion element of the photoelectric conversion elements that outputs the image pickup signal of the infrared component, and
Real (λ) denotes a spectral sensitivity with which a observation target substance used to grasp change in a human body can be picked up with the highest contrast.