Systems and methods for transparent object segmentation using polarization cues

1. A computer-implemented method for computing a prediction on images of a scene, the method comprising:

receiving one or more polarization raw frames of a scene, the polarization raw frames being captured with a polarizing filter at a different linear polarization angle;

extracting one or more first tensors in one or more polarization representation spaces from the polarization raw frames; and

computing a prediction regarding one or more optically challenging objects in the scene based on the one or more first tensors in the one or more polarization representation spaces,

wherein the one or more first tensors in the one or more polarization representation spaces comprise:

a degree of linear polarization (DOLP) image in a DOLP representation space; and

an angle of linear polarization (AOLP) image in an AOLP representation space, and

wherein the computing the prediction comprises supplying the one or more first tensors in the one or more polarization representation spaces, including the AOLP image in the AOLP representation space, to a statistical model.

2. The computer-implemented method of claim 1 , wherein the one or more first tensors further comprise one or more non-polarization tensors in one or more non-polarization representation spaces, and

wherein the one or more non-polarization tensors comprise one or more intensity images in intensity representation space.

3. The computer-implemented method of claim 2 , wherein the one or more intensity images comprise:

a first color intensity image;

a second color intensity image; and

a third color intensity image.

4. The computer-implemented method of claim 1 , wherein the prediction comprises a segmentation mask.

5. The computer-implemented method of claim 1 , wherein the computing the prediction comprises supplying the one or more first tensors to one or more corresponding convolutional neural network (CNN) backbones, and

wherein each of the one or more CNN backbones is configured to compute a plurality of mode tensors at a plurality of different scales.

6. The computer-implemented method of claim 5 , wherein the computing the prediction further comprises:

fusing the mode tensors computed at a same scale by the one or more CNN backbones.

7. The computer-implemented method of claim 6 , wherein the fusing the mode tensors at the same scale comprises:

concatenating the mode tensors at the same scale;

supplying the mode tensors to an attention subnetwork to compute one or more attention maps; and

weighting the mode tensors based on the one or more attention maps to compute a fused tensor for the scale.

8. The computer-implemented method of claim 7 , wherein the computing the prediction further comprises supplying the fused tensors computed at each scale to a prediction module configured to compute the segmentation mask.

9. The computer-implemented method of claim 4 , wherein the segmentation mask is supplied to a controller of a robot picking arm.

10. The computer-implemented method of claim 1 , wherein the prediction comprises a classification of the one or more polarization raw frames based on the one or more optically challenging objects.

11. The computer-implemented method of claim 1 , wherein the prediction comprises one or more detected features of the one or more optically challenging objects depicted in the one or more polarization raw frames.

12. The computer-implemented method of claim 1 ,

wherein the statistical model is trained using training data comprising training first tensors in the one or more polarization representation spaces and labels.

13. The computer-implemented method of claim 12 , wherein the training data comprises:

source training first tensors, in the one or more polarization representation spaces, computed from data captured by a polarization camera; and

additional training first tensors generated from the source training first tensors through affine transformations comprising a rotation.

14. The computer-implemented method of claim 13 , wherein, when the additional training first tensors comprise an angle of linear polarization (AOLP) image, generating the additional training first tensors comprises:

rotating the additional training first tensors by an angle; and

counter-rotating pixel values of the AOLP image by the angle.

15. A computer vision system comprising:

a polarization camera comprising a polarizing filter; and

a processing system comprising a processor and memory storing instructions that, when executed by the processor, cause the processor to:

receive one or more polarization raw frames of a scene, the polarization raw frames being captured with a polarizing filter at a different linear polarization angle;

extract one or more first tensors in one or more polarization representation spaces from the polarization raw frames; and

compute a prediction regarding one or more optically challenging objects in the scene based on the one or more first tensors in the one or more polarization representation spaces,

wherein the one or more first tensors in the one or more polarization representation spaces comprise:

a degree of linear polarization (DOLP) image in a DOLP representation space; and

an angle of linear polarization (AOLP) image in an AOLP representation space, and

wherein the instructions to compute the prediction comprise instructions that, when executed by the processor, cause the processor to supply the one or more first tensors, including the AOLP image in the AOLP representation space, to a statistical model.

16. The computer vision system of claim 15 , wherein the one or more first tensors further comprise one or more non-polarization tensors in one or more non-polarization representation spaces, and

wherein the one or more non-polarization tensors comprise one or more intensity images in intensity representation space.

17. The computer vision system of claim 16 , wherein the one or more intensity images comprise:

a first color intensity image;

a second color intensity image; and

a third color intensity image.

18. The computer vision system of claim 15 , wherein the prediction comprises a segmentation mask.

19. The computer vision system of claim 18 , wherein the memory further stores instructions that, when executed by the processor, cause the processor to compute the prediction by supplying the one or more first tensors to one or more corresponding convolutional neural network (CNN) backbones, and

wherein each of the one or more CNN backbones is configured to compute a plurality of mode tensors at a plurality of different scales.

20. The computer vision system of claim 19 , wherein the memory further stores instructions that, when executed by the processor, cause the processor to:

fuse the mode tensors computed at a same scale by the one or more CNN backbones.

21. The computer vision system of claim 18 , wherein the segmentation mask is supplied to a controller of a robot picking arm.

22. The computer vision system of claim 15 , wherein the prediction comprises a classification of the one or more polarization raw frames based on the one or more optically challenging objects.

23. The computer vision system of claim 15 , wherein the prediction comprises one or more detected features of the one or more optically challenging objects depicted in the one or more polarization raw frames.

24. The computer vision system of claim 15 ,

wherein the statistical model is trained using training data comprising training first tensors in the one or more polarization representation spaces and labels.

25. The computer vision system of claim 24 , wherein the training data comprises:

source training first tensors computed from data captured by a polarization camera; and

additional training first tensors generated from the source training first tensors through affine transformations comprising a rotation.

26. The computer vision system of claim 25 , wherein, when the additional training first tensors comprise an angle of linear polarization (AOLP) image, generating the additional training first tensors comprises:

rotating the additional training first tensors by an angle; and

counter-rotating pixel values of the AOLP image by the angle.

27. A computer vision system comprising:

a polarization camera comprising a polarizing filter; and

a processing system comprising a processor and memory storing instructions that, when executed by the processor, cause the processor to:

receive one or more polarization raw frames of a scene, the polarization raw frames being captured with a polarizing filter at a different linear polarization angle;

extract one or more first tensors in one or more polarization representation spaces from the polarization raw frames; and

compute a prediction regarding one or more optically challenging objects in the scene based on the one or more first tensors in the one or more polarization representation spaces,

wherein the prediction comprises a segmentation mask,

wherein the memory further stores instructions that, when executed by the processor, cause the processor to compute the prediction by supplying the one or more first tensors to one or more corresponding convolutional neural network (CNN) backbones,

wherein each of the one or more CNN backbones is configured to compute a plurality of mode tensors at a plurality of different scales,

wherein the memory further stores instructions that, when executed by the processor, cause the processor to:

fuse the mode tensors computed at a same scale by the one or more CNN backbones, and

wherein the instructions that cause the processor to fuse the mode tensors at the same scale comprise instructions that, when executed by the processor, cause the processor to:

concatenate the mode tensors at the same scale;

supply the mode tensors to an attention subnetwork to compute one or more attention maps; and

weight the mode tensors based on the one or more attention maps to compute a fused tensor for the scale.

28. The computer vision system of claim 27 , wherein the instructions that cause the processor to compute the prediction further comprise instructions that, when executed by the processor, cause the processor to supply the fused tensors computed at each scale to a prediction module configured to compute the segmentation mask.