System and method for joint image refinement and perception

1. A system for end-to-end differentiable joint image refinement and perception, comprising:

a processor;

a learning machine, having a memory having computer readable instructions stored thereon for execution by the processor, causing the processor to:

determine a representation of a raw image of a plurality of raw images, comprising:

transforming pixel-values of the raw image to produce a transformed image comprising transformed pixels of variance-stabilized values;

successively performing processes of:

soft camera projection; and

image projection;

and

inverse transforming the transformed pixels;

initialize a plurality of representation parameters of the representation;

define a plurality of analysis parameters of an image analysis network processing the representation; and

jointly train the plurality of representation parameters and the plurality of analysis parameters to optimize a combined objective function, thereby producing a learned machine.

2. The system of claim 1 , further comprising an image acquisition module for acquiring the plurality of raw images.

3. The system of claim 1 , wherein the computer readable instructions further cause the processor to:

update the plurality of raw images and evaluate the learned machine using an updated plurality of raw images; and

revise the plurality of representation parameters based on results of evaluation.

4. The system of claim 1 , further comprising a learning depot comprising training data and learned data.

5. The system of claim 1 , wherein the image refinement comprises at least one of the following:

demosaicing;

denoising;

deblurring;

tone mapping.

6. The system of claim 1 , wherein the perception comprises image classification.

7. The system of claim 1 , wherein the processor readable instructions further cause the processor to:

implement an Anscombe transformation for the transforming pixel-values; and

implement an unbiased inverse Anscombe transformation for the inverse transforming, thereby producing a learned machine.

8. The system of claim 7 , wherein the processor readable instructions further cause the processor to generate an additional channel to the transformed image.

9. The system of claim 7 , wherein the processor readable instructions causing the image projection comprise computer readable instructions to perform multi-level spatial convolution, pooling, subsampling, and interpolation.

10. The system of claim 9 , wherein the plurality of representation parameters comprises a number of levels for the multi-level spatial convolution, a pooling parameter, a stride of the subsampling, and a step of the interpolation.

11. The system of claim 9 , wherein the processor readable instructions further cause the processor to:

evaluate a performance using a plurality of test images; and

revise the multi-level spatial convolution, the pooling, the subsampling, and the interpolation according to a result of evaluating the performance.

12. A learning machine for joint image refinement and perception, comprising:

a memory having computer readable instructions stored thereon for execution by a processor, forming:

means for determining a representation of a raw image of a plurality of raw images, comprising:

means for transforming pixel-values of the raw image to produce a transformed image comprising transformed pixels of variance-stabilized values;

means for successively performing:

soft camera projection; and

image projection;

and

means for inverse transforming the transformed pixels;

means for initializing a plurality of representation parameters of the representation;

means for defining a plurality of analysis parameters of an image analysis network processing the representation; and

means for jointly training the plurality of representation parameters and the plurality of analysis parameters to optimize a combined objective function, thereby producing a learned machine.

13. The learning machine of claim 12 , further comprising:

means for updating the plurality of raw images and evaluating a performance of the learned machine using an updated plurality of raw images; and

means for revising the plurality of representation parameters based on the evaluating.

14. The learning machine of claim 12 , wherein the image refinement comprises at least one of the following:

demosaicing;

denoising;

deblurring;

tone mapping.

15. The learning machine of claim 12 , wherein the perception comprises image classification.

16. The learning machine of claim 12 , wherein the means for transforming pixel-values further comprise:

means for performing an Anscombe transformation for the transforming pixel-values; and

means for performing an unbiased inverse Anscombe transformation for the inverse transforming.

17. The learning machine of claim 16 , further comprising means for generating an additional channel to the transformed image.

18. The learning machine of claim 16 , wherein the image projection comprises multi-level spatial convolution, pooling, subsampling, and interpolation.

19. The learning machine of claim 18 , wherein the plurality of representation parameters comprises a number of levels for the multi-level spatial convolution, a pooling parameter, a stride of the subsampling, and a step of the interpolation.

20. The learning machine of claim 18 , further comprising:

means for evaluating a performance of the learning machine using a plurality of test images; and

means for revising the multi-level spatial convolution, the pooling, the subsampling, and the interpolation based on the evaluating.

21. A method of machine learning, comprising:

employing a hardware processor for joint image refinement and perception, comprising:

determining a representation of a raw image of a plurality of raw images, comprising:

transforming pixel-values of the raw image to produce a transformed image comprising transformed pixels of variance-stabilized values;

successively performing:

soft camera projection; and

image projection;

and

inverse transforming the transformed pixels;

initializing a plurality of representation parameters of the representation;

defining a plurality of analysis parameters of an image analysis network processing the representation; and

jointly training the plurality of representation parameters and the plurality of analysis parameters to optimize a combined objective function;

thereby producing a learned machine.