System and method for end-to-end-differentiable joint image refinement and perception

1. A system for end-to-end differentiable joint image refinement and perception, comprising:

a processor;

a learning machine, having a memory having computer readable instructions stored thereon for execution by the processor, causing the processor to:

determine a representation of a raw image of said plurality of raw images;

initialize a plurality of representation parameters of said representation;

define a plurality of analysis parameters of an image analysis network configured to process said representation; and

jointly train said plurality of representation parameters and said plurality of analysis parameters to optimize a combined objective function, comprising joint optimization of nested bilevel objective functions, thereby enabling formulation of said combined objective function as an outer objective function relevant to said image analysis network and an inner objective function relevant to said representation.

2. The system of claim 1 , further comprising an image acquisition module for acquiring the plurality of raw images.

3. The system of claim 1 , wherein the computer readable instructions further cause the processor to:

update the plurality of raw images and evaluate the learned machine using an updated plurality of raw images; and

revise the plurality of representation parameters based on results of evaluation.

4. The system of claim 1 , further comprising a learning depot comprising training data and learned data.

5. The system of claim 1 , wherein the image refinement comprises at least one of the following:

demosaicing;

denoising;

deblurring;

tone mapping.

6. The system of claim 1 , wherein the perception comprises image classification.

7. The system of claim 1 , wherein the processor readable instructions further cause the processor to:

implement an Anscombe transformation for the transforming pixel-values; and

implement an unbiased inverse Anscombe transformation for the inverse transforming.

8. The system of claim 7 , wherein the processor readable instructions further cause the processor to generate an additional channel to the transformed image.

9. The system of claim 7 , wherein the processor readable instructions causing the image projection comprise computer readable instructions to perform multi-level spatial convolution, pooling, subsampling, and interpolation.

10. The system of claim 9 , wherein the plurality of representation parameters comprises a number of levels for the multi-level spatial convolution, a pooling parameter, a stride of the subsampling, and a step of the interpolation.

11. The system of claim 9 , wherein the processor readable instructions further cause the processor to:

evaluate a performance using a plurality of test images; and

revise the multi-level spatial convolution, the pooling, the subsampling, and the interpolation according to a result of evaluating the performance.

12. A learning machine for joint image refinement and perception, comprising:

a memory device having processor executable instructions stored thereon for execution by a hardware processor, forming:

means for determining a representation of a raw image of said plurality of raw images;

means for initializing a plurality of representation parameters of said representation;

means for defining a plurality of analysis parameters of an image analysis network configured to process said representation; and

means for jointly training said plurality of representation parameters and said plurality of analysis parameters to optimize a combined objective function, comprising jointly optimizing of nested bilevel objective functions, thereby enabling formulation of said combined objective function as an outer objective function relevant to said image analysis network and an inner objective function relevant to said representation.

13. The learning machine of claim 12 , further comprising:

means for updating the plurality of raw images and evaluating a performance of the learned machine using an updated plurality of raw images; and

means for revising the plurality of representation parameters based on the evaluating.

14. The learning machine of claim 12 , wherein the image refinement comprises at least one of the following:

demosaicing;

denoising;

deblurring;

tone mapping.

15. The learning machine of claim 12 , wherein the perception comprises image classification.

16. The learning machine of claim 12 , wherein the means for determining further comprise:

means for performing an Anscombe transformation for transforming pixel-values of the raw image; and

means for performing an unbiased inverse Anscombe transformation.

17. The learning machine of claim 16 , further comprising means for generating an additional channel to the transformed image.

18. The learning machine of claim 16 , wherein the image projection comprises multi-level spatial convolution, pooling, subsampling, and interpolation.

19. The learning machine of claim 18 , wherein the plurality of representation parameters comprises a number of levels for the multi-level spatial convolution, a pooling parameter, a stride of the subsampling, and a step of the interpolation.

20. The learning machine of claim 18 , further comprising:

means for evaluating a performance of the learning machine using a plurality of test images; and

means for revising the multi-level spatial convolution, the pooling, the subsampling, and the interpolation based on the evaluating.

21. A method of machine learning, comprising:

employing a hardware processor for joint image refinement and perception, comprising:

determining a representation of a raw image of said plurality of raw images;

initializing a plurality of representation parameters of said representation;

defining a plurality of analysis parameters of an image analysis network configured to process said representation; and

jointly training said plurality of representation parameters and said plurality of analysis parameters to optimize a combined objective function, comprising formulating said combined objective function as a nested bilevel objective function comprising an outer objective function relevant to said image analysis network and an inner objective function relevant to said representation;

thereby producing a learned machine.