Permutation invariant high dynamic range imaging
An image processing apparatus for forming an enhanced image is disclosed. The apparatus comprises one or more processors configured to: receive one or more input images; form, from each of the one or more input images, a respective feature representation, each feature representation representing features of the respective input image; and subject the one or more feature representations to a symmetric pooling operation to form an enhanced image from at least some of the features of the one or more feature representations identified by the symmetric pooling operation. The apparatus may generate images with increased photoreceptive dynamic range, increased bit depth and signal-to-noise ratio, with less quantization error and richer colour representation.
1 . An image processing apparatus for forming an enhanced image, comprising one or more processors configured to:
receive one or more input images;
form, from each of the one or more input images, a respective feature representation, each feature representation representing features of the respective input image; and
subject the one or more feature representations to a symmetric pooling operation to form an enhanced image from at least some of the features of the one or more feature representations identified by the symmetric pooling operation,
wherein the symmetric pooling operation is performed in conjunction with a trained neural network,
wherein the enhanced image is an estimated high-dynamic-range image and the one or more input images are of lower dynamic range than the estimated high-dynamic-range image, and
wherein the symmetric pooling operation operates independently of an order in which the feature representations are provided to the symmetric pooling operation.
2 . An image processing apparatus as claimed in claim 1 , wherein the symmetric pooling operation is performed to subject each feature representation to a common processing logic so as to identify components of the respective feature representation.
3 . An image processing apparatus as claimed in claim 1 , wherein the symmetric pooling operation comprises a max pooling operation and/or an average pooling operation.
4 . An image processing apparatus as claimed in claim 1 , wherein the one or more processors are configured to:
subject each input image or data derived from the input image to a first convolution operation to form the respective feature representation(s); and
form the enhanced image by subjecting an output of the symmetric pooling operation to a second convolution operation.
5 . An image processing apparatus as claimed in claim 4 , wherein the one or more processors are configured to:
subject an output of the symmetric pooling operation and the respective feature representation to a concatenation operation to form a respective intermediate feature representation; and
wherein the second convolution operation is performed on the intermediate feature representations.
6 . An image processing apparatus as claimed in claim 1 , wherein the one or more processors are configured to form each feature representation by performing one or more of refinement tasks including: denoising, dequantization, linearisation, camera response calibration and over and under-exposure correction on the respective input image.
7 . An image processing apparatus as claimed in claim 1 , comprising a camera and wherein the one or more processors are configured to receive the input images from the camera.
8 . An image processing apparatus as claimed in claim 7 , wherein the camera is configured to capture the input images at different times and with different exposure settings.
9 . An image processing apparatus as claimed in claim 1 , wherein the one or more input images are subject to exposure normalization.
10 . An image processing apparatus as claimed in claim 1 , wherein the apparatus is configured to receive a single input image.
11 . An image processing apparatus as claimed in claim 1 , wherein the enhanced image is formed by fusing at least some of the features of the one or more feature representations.
12 . A computer-implemented method for forming an enhanced image, comprising:
receiving one or more input images;
forming, from each of the one or more input images, a respective feature representation, each feature representation representing features of the respective input image; and
subjecting the one or more feature representations to a symmetric pooling operation to form an enhanced image from at least some of the features of the feature representations identified by the symmetric pooling operation,
wherein the symmetric pooling operation is performed in conjunction with a trained neural network,
wherein the enhanced image is an estimated high-dynamic-range image and the one or more input images are of lower dynamic range than the estimated high dynamic range image, and
wherein the symmetric pooling operation operates independently of an order in which the feature representations are provided to the symmetric pooling operation.
13 . A method as claimed in claim 12 , wherein the receiving of one or more input images comprises receiving multiple input images and fields of view of the multiple input images are the same.
14 . A method as claimed in claim 12 , wherein the receiving of one or more input images comprises receiving multiple input images and fields of view of at least two of the multiple input images are different.
15 . A method as claimed in claim 12 , wherein the symmetric pooling operation is performed to subject each feature representation to a common processing logic so as to identify components of the respective feature representation.