Deep-learning method for separating reflection and transmission images visible at a semi-reflective surface in a computer image of a real-world scene

1. A method, comprising:

synthesizing training data for a deep learning network by applying a plurality of manipulations to a training reflection image and a training transmission image for a semi-reflective surface representation, the plurality of manipulations simulating behaviors observed in real-world data;

training the deep learning network to learn a residual representation of a reflection and transmission relative to input images, using the training data;

receiving, as input to the deep learning network, polarization images of a real-world scene having a semi-reflective surface;

outputting, by the deep learning network, the residual representation of the reflection and transmission for the semi-reflective surface of the real-world scene.

2. The method of claim 1 , wherein the training reflection image and the training transmission image are included in a data point randomly selected from a training image set.

3. The method of claim 1 , wherein the plurality of manipulations are applied to the training reflection image and training transmission image through a data generation pipeline that takes the training reflection image and training transmission image as input and that outputs the training data for the deep learning network.

4. The method of claim 1 , wherein the plurality of manipulations includes:

manipulating the dynamic range (DR) of the training reflection image and the training transmission image.

5. The method of claim 4 , wherein manipulating the DR of the training reflection image and the training transmission image includes brightening the training reflection image or the training transmission image.

6. The method of claim 4 , wherein manipulating the DR of the training reflection image and the training transmission image includes providing edge-aware reflection.

7. The method of claim 6 , wherein the edge-aware reflection is provided by setting to zero regions of the training reflection image having an intensity below a defined threshold.

8. The method of claim 1 , wherein the plurality of manipulations includes:

manipulating the training reflection image and the training transmission image to simulate artifacts caused by movement.

9. The method of claim 8 , wherein the artifacts include non-rigid deformations.

10. The method of claim 8 , wherein the artifacts are simulated by defining a grid over a patch of the training reflection image, perturbing the grid's anchors by a selected x,y amount, and interpolating a position of remaining pixels in the patch.

11. The method of claim 10 , wherein polarization images are created for the patch, each of the polarization images being a separate image created for a different polarization angle.

12. The method of claim 1 , wherein the plurality of manipulations includes:

manipulating the training reflection image and the training transmission image to simulate local curvatures of the semi-reflective surface representation.

13. The method of claim 12 , wherein the local curvatures are simulated using a parabola by sampling four parameters: a camera position, a point on the semi-reflective surface representation, a segment length, and a convexity as +/−1.

14. The method of claim 1 , wherein the training data includes:

a latent reflection image,

a latent transmission image, and

training polarization images.

15. The method of claim 1 , wherein the polarization images captured for the scene include a plurality of images of the scene captured at different polarization angles.

16. The method of claim 1 , wherein the deep learning network generates an estimated reflection image and an estimated transmission image from the polarization images.

17. The method of claim 16 , wherein the deep learning network learns the residual representation of the reflection and transmission for the semi-reflective surface of the real-world scene, using the estimated reflection image and the estimated transmission image.

18. A non-transitory computer readable storing code executable by a processor to perform a method comprising:

synthesizing training data for a deep learning network by applying a plurality of manipulations to a training reflection image and a training transmission image for a semi-reflective surface representation, the plurality of manipulations simulating behaviors observed in real-world data;

training the deep learning network to learn a residual representation of a reflection and transmission relative to input images, using the training data;

receiving, as input to the deep learning network, polarization images of a real-world scene having a semi-reflective surface;

outputting, by the deep learning network, the residual representation of the reflection and transmission for the semi-reflective surface of the real-world scene.

19. A system, comprising:

a memory; and

at least one processor for:

synthesizing training data for a deep learning network by applying a plurality of manipulations to a training reflection image and a training transmission image for a semi-reflective surface representation, the plurality of manipulations simulating behaviors observed in real-world data;

training the deep learning network to learn a residual representation of a reflection and transmission relative to input images, using the training data;

receiving, as input to the deep learning network, polarization images of a real-world scene having a semi-reflective surface;

outputting, by the deep learning network, the residual representation of the reflection and transmission for the semi-reflective surface of the real-world scene.