Methods and systems for super resolution for infra-red imagery
An artificial neural network for processing low-resolution images to generate super-resolution images includes feed-forward connections between layers. The network includes an input layer, one or more convolution layers, wherein the input layer is connected to a first convolution layer of the convolution layers, and a output layer connected to a last convolution layer of the convolution layers. An input image is input to the input layer and to at least one of the convolution layers, an initial output of the input layer is input to at least one of the convolution layers, and a layer output of at least one of the convolution layers is input to at least one subsequent convolution layer.
1. A system for image processing, the system comprising
a processing system containing one or more processors, and
an artificial neural network ( 100 ), the latter comprising:
(i) an input layer (L-IN) including a memory location for storing an input image (ILR),
(ii) one or more (L) convolution layers (L-n), wherein said input layer (L-IN) is connected to a first convolution layer (L-1) of said convolution layers (L-n), and
(iii) an output layer (L-OUT) connected to a last convolution layer (L-l) of said convolution layers (L-n) and including a memory location for storing an output image (ISR), said output layer (L-OUT) comprising:
(a) a shuffleblock (L-SB) receiving a layer output (Sl) of said last convolution layer (L-l) and the input image (ILR) and generating a shuffle-block output ( 114 ) that is a higher resolution than the input image (ILR) and said layer output (SI),
(b) an interpolation module ( 112 ) receiving the input image (ILR) and generating an interpolated image ( 116 ) that is higher resolution than the input image (ILR), and
(c) a final convolution (L-FIN) receiving said shuffle-block output ( 114 ) and said interpolated image ( 116 ) and generating said output image (ISR),
wherein
(A) said layers include instructions for execution on said processing system,
(B) the input image (ILR) is input to said input layer (L-IN) and to at least one of said convolution layers (L-n),
(C) an initial output ( 110 ) of said input layer (L-IN) is input to at least one of said convolution layers (L-n), and
(D) a layer output(S) of at least one of said convolution layers (L-n) is input to at least one subsequent convolution layer (L-n).
2. The system of claim 1 wherein the processors are configured to execute instructions programmed using a predefined set of machine codes and said layers include computational instructions implemented in the machine codes of the processor.
3. The system of claim 1 wherein said input image is a low-resolution image and said output image is a super-resolution image.
4. The system of claim 1 wherein each of at least one of said convolution layers (L-n), includes:
(a) a respective convolution module (LCON-n) accepting data to respective said convolution layer (L-n),
(b) a respective activation function (PReLU) processing output data from said respective convolution module (LCON-n), and
(c) a respective bottleneck layer (LB) processing output data from said respective activation function (PReLU).
5. The system of claim 4 wherein the input image (ILR) and said initial output ( 110 ) are input to said bottleneck layer (LB), and said bottleneck layer (LB) generates said layer output(S).
6. The system of claim 1 wherein said layer output (Sl) is input to each subsequent convolution layer (L-n).
7. The system of claim 1 wherein said network ( 100 ) is trained with a training set based on high-resolution images and corresponding low-resolution images.
8. A method of training the network ( 100 ) of claim 1 , the method comprising the steps of:
(a) receiving one or more sets of high-resolution (IHR) images,
(b) applying one or more transformations to at least a subset of said sets of high-resolution images to generate at least one associated set of low-resolution images (ILR),
(c) creating a training set including said one or more sets of high-resolution images and said at least one associated set of low-resolution images, and
(d) training ( 120 ) the network ( 100 ) using said training set.
9. A method for image processing, the method comprising the steps of:
(i) configuring an artificial neural network ( 100 ) based on a training set of high-resolution images and corresponding low-resolution images, and
(ii) inputting an input image (ILR) to an input layer (L-IN) and to at least one convolution layer (L-n),
(iii) generating an initial output ( 110 ) from said input layer (L-IN) based on said input image (ILR) and sending said initial output ( 110 ) to at least a first convolutional layer (L-1) of said convolution layers (L-n), and
(iv) generating a current layer output(S) of at least one of said convolution layers (L-n) based on said input image (ILR), said initial output ( 110 ) and any previous layer outputs(S), and sending said current layer output(S) to at least one subsequent convolution layer (L-n), and
(v) generating an output image (ISR) by an output layer (L-OUT) based on a layer output (Sl) of a last convolutional layer (L-l) of said convolutional layers (L-n) and said input image (ILR),
wherein said output layer (L-OUT) comprises:
(a) a shuffleblock (L-SB) receiving the layer output (Sl) of said last convolution layer (L-l) and the input image (ILR) and generating a shuffle-block output ( 114 ) that is a higher resolution than the input image (ILR) and said layer output (Sl),
(b) an interpolation module ( 112 ) receiving the input image (ILR) and generating an interpolated image ( 116 ) that is higher resolution than the input image (ILR), and
(c) a final convolution (L-FIN) receiving said shuffle-block output ( 114 ) and said interpolated image ( 116 ) and generating said output image (ISR).
10. A computer usable non-transitory storage medium having a computer program embodied thereon for causing a suitably programmed system to process images, by performing the steps of claim 9 when such program is executed on the system.