Illumination spectrum recovery
View Patent ↗This disclosure concerns processing of electronic images, such as hyperspectral or multispectral images. In particular, this disclosure provides methods, software and computer systems for estimating an illumination spectrum of a digital image. A processor applies a neural network to the digital image by calculating three-dimensional convolutions in one or more convolutional layers of the neural network. The three-dimensional convolutions comprise a convolution along a spectral dimension. The processor then evaluates an output layer, connected to the one or more convolutional layers in the neural network. The output layer has multiple output values that each provide an intensity value for a respective band of the illumination spectrum of the digital image.
1 . A method for determining an illumination spectrum in a digital image, wherein the digital image is a hyperspectral image, the method comprising:
applying a neural network to the digital image by:
calculating three-dimensional convolutions in one or more convolutional layers of the neural network, the three-dimensional convolutions comprising a convolution along a spectral dimension, and each of the one or more convolutional layers comprises a convolution filter having a depth, a height and a width, wherein the depth of the convolution filter is greater than the height of the convolution filter and greater than the width of the convolution filter; and
evaluating an output layer, connected to the one or more convolutional layers in the neural network, the output layer having multiple output values that each provide an intensity value for a respective band of the illumination spectrum of the digital image.
2 . The method of claim 1 , further comprising training the neural network by applying a smoothing function to the output values of the output layer to calculate a cost value that is to be minimised during training.
3 . The method of claim 2 , wherein the smoothing function comprises a cubic spline approximation to the output values of the output layer.
4 . The method of claim 1 , further comprising down-sampling bands of the digital image, wherein the one or more convolutional layers are configured to down-sample the bands of the digital image.
5 . The method of claim 1 , further comprising up-sampling a result of the convolutional layer.
6 . The method of claim 1 , further comprising training the neural network on multiple training images.
7 . The method of claim 6 , wherein training comprises extracting from the multiple training images an observed illumination spectrum from a white patch in the image, and wherein training further comprises generating multiple sub-images from the multiple training images and minimising an error between the determined illumination spectrum and the observed illumination spectrum for the multiple sub-images.
8 . The method of claim 7 , wherein the error is based on a cubic smoothing spline function.
9 . The method of claim 8 , wherein the error is represented by an error function comprising a first summand based on a mean square error and a second summand representing a roughness penalty.
10 . The method of claim 9 , wherein the roughness penalty is based on a forward difference of output values.
11 . The method of claim 1 , wherein the neural network is based on ResNet.
12 . The method of claim 1 , wherein the output layer is a fully connected layer.
13 . The method of claim 1 , further comprising processing the digital image based on the illumination spectrum.
14 . The method of claim 13 , wherein processing the digital image comprises calculating a reflectance image by normalising the digital image in relation to the illumination spectrum.
15 . The method of claim 1 , wherein the neural network comprises a max pooling layer, the max pooling layer comprising a filter having a size that is smaller than the convolution filter.
16 . A non-transitory, computer readable medium with program code stored thereon that, when executed by a computer, causes the computer to perform the method of claim 1 .
17 . A computer system for determining an illumination spectrum in a digital image, wherein the digital image is a hyperspectral image, the computer system comprising a processor configured to apply a neural network to the digital image by:
calculating three-dimensional convolutions in one or more convolutional layers of the neural network, the three-dimensional convolutions comprising a convolution along a spectral dimension, and each of the one or more convolutional layers comprises a convolution filter having a depth, a height and a width, wherein the depth of the convolution filter is greater than the height of the convolution filter and greater than the width of the convolution filter; and
evaluating an output layer, connected to the one or more convolutional layers in the neural network, the output layer having multiple output values that each provide an intensity value for a respective band of the illumination spectrum of the digital image.
18 . The computer system of claim 17 , further comprising an image sensor to generate the digital image and a storage medium to store the digital image and the illumination spectrum.