Systems and methods for augmentation of sensor systems and imaging systems with polarization
A multi-modal sensor system includes: an underlying sensor system; a polarization camera system configured to capture polarization raw frames corresponding to a plurality of different polarization states; and a processing system including a processor and memory, the processing system being configured to control the underlying sensor system and the polarization camera system, the memory storing instructions that, when executed by the processor, cause the processor to: control the underlying sensor system to perform sensing on a scene and the polarization camera system to capture a plurality of polarization raw frames of the scene; extract first tensors in polarization representation spaces based on the plurality of polarization raw frames; and compute a characterization output based on an output of the underlying sensor system and the first tensors in polarization representation spaces.
1. A multi-modal sensor system comprising:
an underlying sensor system;
a polarization camera system configured to capture polarization raw frames corresponding to a plurality of different polarization states; and
a processing system comprising a processor and memory, the processing system being configured to control the underlying sensor system and the polarization camera system, the memory storing instructions that, when executed by the processor, cause the processor to:
control the underlying sensor system to perform sensing on a scene and the polarization camera system to capture a plurality of polarization raw frames of the scene;
extract first tensors in polarization representation spaces based on the plurality of polarization raw frames; and
compute a characterization output based on an output of the underlying sensor system and the first tensors in polarization representation spaces.
2. The multi-modal sensor system of claim 1 , wherein the polarization camera system comprises a polarization camera module comprising:
a first polarization camera comprising a first polarizing filter at a first polarization orientation, the first polarization camera having a first optical axis;
a second polarization camera comprising a second polarizing filter at a second polarization orientation, the second polarization camera having a second optical axis substantially parallel to the first optical axis; and
a third polarization camera comprising a third polarizing filter at a third polarization orientation, the third polarization camera having a third optical axis substantially parallel to the first optical axis.
3. The multi-modal sensor system of claim 2 , wherein the polarization camera module further comprises a fourth polarization camera comprising a fourth polarizing filter at a fourth polarization orientation, the fourth polarization camera having a fourth optical axis substantially parallel to the first optical axis.
4. The multi-modal sensor system of claim 2 , wherein the first tensors comprise a degree of linear polarization (DOLP) and an angle of linear polarization (AOLP), and
wherein the memory further stores instructions that, when executed by the processor, cause the processor to compute the DOLP and the AOLP based on polarization raw frames captured by the first polarization camera, the second polarization camera, and the third polarization camera, the instructions comprising instructions to:
initialize an estimated DOLP and an estimated AOLP based on Stokes vectors;
estimate a scene geometry based on parallax shifts in the polarization raw frames to generate a coarse model; and
iteratively:
refine the coarse model based on the estimated DOLP and the estimated AOLP to generate an estimated geometry; and
update the estimated DOLP and the estimated AOLP based on the estimated geometry,
until a change in the estimated DOLP and a change in the estimated AOLP are both less than corresponding threshold values.
5. The multi-modal sensor system of claim 1 , wherein the polarization camera system comprises a stereo polarization camera system comprising:
a first polarization camera module having a first optical axis, the first polarization camera module being configured to capture a first plurality of polarization raw frames corresponding to a first plurality of different polarization states; and
a second polarization camera module having a second optical axis and spaced apart from the first polarization camera module along a baseline, the second polarization camera module being configured to capture a second plurality of polarization raw frames corresponding to a second plurality of different polarization states,
the first optical axis being substantially parallel to the second optical axis.
6. The multi-modal sensor system of claim 5 , wherein the first polarization camera module comprises a first plurality of color filters configured to transmit light in three or more different first color spectra, and
wherein the second polarization camera module comprises a second plurality of color filters configured to transmit light in three or more different second color spectra,
wherein the three or more second color spectra are different from the three or more first color spectra.
7. The multi-modal sensor system of claim 6 , wherein the memory further stores instructions that, when executed by the processor, cause the processor to:
control the stereo polarization camera system to capture multi-spectral stereo polarization imaging data in the first color spectra and in the second color spectra; and
extract first tensors in polarization representation spaces from the multi-spectral stereo polarization imaging data.
8. The multi-modal sensor system of claim 1 , wherein the underlying sensor system comprises an active scanning system comprising an active emitter and a detector.
9. The multi-modal sensor system of claim 8 , wherein the active scanning system comprises a radar system.
10. The multi-modal sensor system of claim 8 , wherein the active scanning system comprises a lidar system.
11. The multi-modal sensor system of claim 8 , wherein the active scanning system comprises an active stereo depth camera system.
12. The multi-modal sensor system of claim 8 , wherein the multi-modal sensor system is mounted on a vehicle.
13. The multi-modal sensor system of claim 8 , wherein the memory further stores instructions that, when executed by the processor of the processing system, cause the processor to:
compute a sparse point cloud based on the output of the underlying sensor system;
compute surface normals from the polarization raw frames;
compute a 3-D surface based on the surface normals; and
correct the 3-D surface based on the sparse point cloud to compute a 3-D model of the scene.
14. The multi-modal sensor system of claim 8 , wherein the memory further stores instructions that, when executed by the processor of the processing system, cause the processor to:
compute a segmentation map based on the first tensors in the polarization representation spaces, the segmentation map identifying one or more regions of interest in the scene;
steer the active emitter to emit beams toward the one or more regions of interest; and
detect a reflection of the beams emitted by the active emitter using the detector of the active scanning system.
15. The multi-modal sensor system of claim 1 , wherein the memory stores instructions that cause the processor to implement a convolutional neural network trained to compute a segmentation map based on the first tensors.
16. The multi-modal sensor system of claim 1 , wherein the underlying sensor system comprises a color camera system.
17. The multi-modal sensor system of claim 16 , wherein the color camera system is a digital single lens reflex camera or a video camera.
18. The multi-modal sensor system of claim 16 , wherein the output of the color camera of the underlying sensor system comprises a color image, and
wherein the memory further stores instructions that, when executed by the processor, cause the processor to:
compute a plurality of surface normals of the scene based on the first tensors; and
store the computed surface normals of the scene in a same file as the color image captured by the color camera.