Polarized image enhancement using deep neural networks

1 . A method for processing a polarized image, comprising:

determining, by a computing device, a polarization characterization for a polarized image, wherein the polarization characterization is indicative of polarization data associated with a plurality of polarization directions of incident light in the polarized image;

extracting, by the computing device and from the polarized image, a first collection of global features and a second collection of local features, wherein the extraction is performed using a guided filtering technique;

performing, by the computing device and based on the polarization characterization:

a global feature fusion to fuse global features in the first collection, and

a local feature fusion to fuse local features in the second collection by utilizing a neural network to adaptively fuse an image intensity associated with the polarized image and the polarization characterization for the polarized image;

compositing, by the computing device, the polarization characterization with the fused global features and the fused local features to generate a reconstructed image; and

providing, by the computing device, the reconstructed image to an image processing resource to perform one or more image processing tasks.

2 . The method of claim 1 , further comprising:

training the neural network with training data adapted to the one or more image processing tasks.

3 . The method of claim 1 , further comprising:

providing the image intensity associated with the polarized image and the polarization characterization for the polarized image; and

receiving the fused local features.

4 . The method of claim 1 , wherein the performing of the global feature fusion comprises:

applying a weighted average fusing to the global features in the first collection to preserve structural information in the polarized image.

5 . The method of claim 1 , wherein the determining of the polarization characterization further comprises:

generating, for the plurality of polarization directions, a corresponding plurality of channel images, by applying a channel splitting operation, wherein the plurality of channel images represent light intensities corresponding to the plurality of polarization directions; and

generating, based on the plurality of channel images, a degree of linear polarization (DoLP) image.

6 . The method of claim 1 , further comprising:

receiving, by the computing device, the polarized image from a polarization sensor configured to collect the incident light at the plurality of polarization directions.

7 . The method of claim 6 , wherein the polarization sensor is at least one of a component of a polarization camera or configured to be removably attached to a camera.

8 . The method of claim 6 , wherein the computing device comprises a polarization camera, and wherein the polarization sensor is a component of the polarization camera.

9 . The method of claim 6 , further comprising:

recommending, based on the reconstructed image, an adjustment to an automatic image capture setting for a camera associated with the polarization sensor.

10 . The method of claim 1 , wherein the one or more image processing tasks comprises of one or more of object detection or object recognition in the reconstructed image.

11 . The method of claim 1 , wherein the plurality of polarization directions are determined based on the one or more image processing tasks.

12 . A computing device for processing a polarized image, comprising:

one or more processors; and

data storage, wherein the data storage has stored thereon computer-executable instructions that, when executed by the one or more processors, cause the computing device to perform operations comprising:

receiving, by the computing device, the polarized image from a polarization sensor configured to collect incident light at a plurality of polarization directions;

determining, by the computing device, a polarization characterization for the polarized image, wherein the polarization characterization is indicative of polarization data associated with the plurality of polarization directions of the incident light in the polarized image;

extracting, by the computing device and from the polarized image, a first collection of global features and a second collection of local features;

performing, by the computing device and based on the polarization characterization:

a global feature fusion to fuse global features in the first collection, and

a local feature fusion to fuse local features in the second collection;

compositing, by the computing device, the polarization characterization with the fused global features and the fused local features to generate a reconstructed image;

providing, by the computing device, the reconstructed image to an image processing resource to perform one or more image processing tasks; and

recommending, based on the reconstructed image, an adjustment to an automatic image capture setting for a camera associated with the polarization sensor.

13 . The computing device of claim 12 , wherein the operations for the performing of the local feature fusion further comprise:

utilizing a neural network to adaptively fuse an image intensity associated with the polarized image and the polarization characterization for the polarized image.

14 . The computing device of claim 13 , the operations further comprising:

training the neural network with training data adapted to the one or more image processing tasks.

15 . The computing device of claim 13 , wherein the operations for the determining of the polarization characterization further comprise:

generating, for the plurality of polarization directions, a corresponding plurality of channel images, by applying a channel splitting operation, wherein the plurality of channel images represent light intensities corresponding to the plurality of polarization directions; and

generating, based on the plurality of plurality of channel images, a degree of linear polarization (DoLP) image.

16 . The computing device of claim 12 , wherein the operations for the performing of the global feature fusion comprise:

applying a weighted average fusing to the global features in the first collection to preserve structural information in the polarized image.

17 . The computing device of claim 12 , wherein the operations for the determining of the polarization characterization further comprise:

generating, for the plurality of polarization directions, a corresponding plurality of channel images, by applying a channel splitting operation, wherein the plurality of channel images represent light intensities corresponding to the plurality of polarization directions; and

generating, based on the plurality of plurality of channel images, a degree of linear polarization (DoLP) image.

18 . The computing device of claim 12 , wherein the polarization sensor is at least one of a component of a polarization camera or configured to be removably attached to a camera.

19 . A method for processing a polarized image, comprising:

determining, by a computing device, a polarization characterization for a polarized image, wherein the polarization characterization is indicative of polarization data associated with a plurality of polarization directions of incident light in the polarized image;

extracting, by the computing device and from the polarized image, a first collection of global features and a second collection of local features;

performing, by the computing device and based on the polarization characterization:

a global feature fusion to fuse global features in the first collection, and

a local feature fusion to fuse local features in the second collection by utilizing a neural network to adaptively fuse an image intensity associated with the polarized image and the polarization characterization for the polarized image, and further comprising:

training the neural network with training data adapted to one or more image processing tasks;

compositing, by the computing device, the polarization characterization with the fused global features and the fused local features to generate a reconstructed image; and

providing, by the computing device, the reconstructed image to an image processing resource to perform the one or more image processing tasks.