Image processing apparatus and image processing method
The present disclosure relates to an image processing apparatus, an image processing method, and a program capable of realizing highly accurate image recognition processing while taking privacy into consideration with a simple configuration. A mask of a lensless camera is convoluted with a weight of a DNN first layer used in image recognition processing using a DNN, and a reconstructed image becomes a processing result of the DNN first layer on the basis of a captured image captured by an image sensor. Accordingly, the restored image becomes the processing result of the DNN first layer and becomes an image difficult for a person to visually recognize as a scene or an object, making it possible to protect the privacy of the image. The present disclosure can be applied to an image recognition apparatus using a lensless camera.
1 . An image processing apparatus, comprising:
a mask configured to modulate and transmit incident light;
an image sensor configured to capture a modulated image based on the incident light modulated by the mask; and
a central processing unit (CPU) configured to perform, in a plurality of channels, a signal processing operation based on a specific mask pattern on the modulated image, wherein
the specific mask pattern for each channel is a pattern obtained as a weighted sum of a binary pattern serving as a basic pattern for each of the plurality of channels and a weight used for the signal processing operation for each of the plurality of channels.
2 . The image processing apparatus according to claim 1 , wherein
the incident light is reflected light reflected from a subject,
the signal processing operation is an image recognition processing operation for the subject using a deep neural network (DNN), and
the weight is a weight of a first layer of the DNN.
3 . The image processing apparatus according to claim 2 , wherein the weight of the first layer of the DNN is learned so that recognition accuracy of the image recognition processing operation becomes higher, and a degree of similarity between an original image captured by the image sensor in a state in which the modulation by the mask is not applied and a restored image obtained by a restoring operation of the modulated image using the weight of the first layer of the DNN becomes lower.
4 . The image processing apparatus according to claim 1 , wherein
the mask is divided into as many sub-areas as a number of channels, and
the specific mask pattern is based on a pattern for each of the plurality of channels in each of the sub-areas.
5 . The image processing apparatus according to claim 4 , wherein
a margin area is a light-shielding area, and
the margin area is between the sub-areas.
6 . The image processing apparatus according to claim 5 , wherein a width of the margin area is based on a field of view (FOV) of a pixel of the image sensor.
7 . The image processing apparatus according to claim 4 , wherein a light shielding wall is between the sub-areas.
8 . The image processing apparatus according to claim 4 , wherein sizes and shapes of the sub-areas are the same.
9 . The image processing apparatus according to claim 4 , wherein sizes and shapes of the sub-areas are not the same.
10 . The image processing apparatus according to claim 4 , wherein basic patterns, of the plurality of channels constituting patterns in the respective sub-areas, have no correlation or a correlation lower than a specific correlation.
11 . The image processing apparatus according to claim 4 , wherein basic patterns, of the plurality of channels constituting patterns in the respective sub-areas, have no correlation or a correlation lower than a specific correlation using a pseudo-random signal.
12 . The image processing apparatus according to claim 1 , wherein a mask pattern is obtained by an overlapping operation of the specific mask pattern for the each of the plurality of channels.
13 . The image processing apparatus according to claim 12 , wherein the basic pattern for the each of the plurality of channels have no correlation or a correlation lower than a specific correlation.
14 . The image processing apparatus according to claim 12 , wherein the basic pattern for the each of the plurality of channels have no correlation or a correlation lower than a specific correlation using a pseudo-random signal.
15 . The image processing apparatus according to claim 1 , wherein
the mask is divided into as many sub-areas as a number of channels,
mask patterns of the plurality of channels are in each of the sub-areas, and
a mask pattern for each sub-area is obtained by an overlapping operation of the mask patterns of the plurality of channels.
16 . The image processing apparatus according to claim 1 , wherein the specific mask pattern consists of a transmission area, a light shielding area, and an intermediate transmissive area.
17 . The image processing apparatus according to claim 1 , wherein the specific mask pattern is a diffraction grating.
18 . An image processing method, comprising:
modulating and transmitting, by a mask, incident light;
capturing, by an image sensor, a modulated image based on the incident light modulated by the mask; and
performing, by a central processing unit (CPU), in a plurality of channels, a signal processing operation based on a specific mask pattern to the modulated image, wherein
the specific mask pattern for each channel is a pattern obtained as a weighted sum of a binary pattern serving as a basic pattern for each of the plurality of channels and a weight used for the signal processing operation for each of the plurality of channels.
19 . A non-transitory computer-readable medium having stored thereon, computer-executable instructions which, when executed by, cause the computer to execute operations, the operations comprising:
modulating and transmitting, by a mask, incident light, and
capturing, by an image sensor, a modulated image based on the incident light modulated by the mask; and
performing, by a central processing unit (CPU), in a plurality of channels, a signal processing operation based on a specific mask pattern on the modulated image, wherein
the specific mask pattern for each channel is a pattern obtained as a weighted sum of a binary pattern serving as a basic pattern for each of the plurality of channels and a weight used for the signal processing operation for each of the plurality of channels.