Imaging sensor, an image processing device and an image processing method
An image processing device has circuitry, which is configured to obtain global image data including image data and spectral data, the global image data representing a global image area and to input the global image data to machine learning model for generating output spectral data, wherein the neural network is configured to transform the obtained image data into the output spectral data based on the obtained spectral data.
1. An image processing device comprising circuitry configured to:
obtain global image data including image data and spectral data, the global image data representing a global image area, wherein the spectral data represents spectral information acquired from a plurality of spectral imaging portions distributed in a photosensitive area of an imaging sensor, and wherein the image data represents imaging information acquired from an imaging part of the photosensitive area;
input the global image data to a convolutional neural network for generating output spectral data, wherein the convolutional neural network is configured to transform the obtained image data into the output spectral data based on the obtained spectral data, wherein the global image area includes more pixels represented by the obtained image data than pixels represented by the obtained spectral data,
wherein the convolutional neural network is trained using:
a back transformation that converts predicted spectral data to back-transformed image data using a physical model that integrates the spectrum according to color sensitivities;
a first loss function computed between the obtained image data and the back-transformed image data;
a second loss function computed between the obtained spectral data and the predicted spectral data; and
a smoothening constraint applied to the predicted spectral data.
2. The image processing device of claim 1 , wherein the obtained spectral data represent spectral information and the obtained image data represent imaging information.
3. The image processing device of claim 1 , wherein the obtained image data are RGB image data and the obtained spectral data are multispectral or hyperspectral data.
4. An image processing method comprising:
obtaining global image data including image data and spectral data, the global image data representing a global image area, wherein the spectral data represents spectral information acquired from a plurality of spectral imaging portions distributed in a photosensitive area of an imaging sensor, and wherein the image data represents imaging information acquired from an imaging part of the photosensitive area;
inputting the global image data to a convolutional neural network for generating output spectral data, wherein the convolutional neural network is configured to transform the obtained image data into the output spectral data based on the obtained spectral data, wherein the global image area includes more pixels represented by the obtained image data than pixels represented by the obtained spectral data,
wherein the convolutional neural network is trained using:
a back transformation that converts predicted spectral data to back-transformed image data using a physical model that integrates the spectrum according to color sensitivities;
a first loss function computed between the obtained image data and the back-transformed image data;
a second loss function computed between the obtained spectral data and the predicted spectral data; and
a smoothening constraint applied to the predicted spectral data.
5. The image processing method of claim 4 , wherein the obtained spectral data represent spectral information and the obtained image data represent imaging information.