Two-stage depth estimation machine learning algorithm and spherical warping layer for EQUI-rectangular projection stereo matching
A system and method is disclosed having an end-to-end two-stage depth estimation deep learning framework that takes one spherical color image and estimate dense spherical depth maps. The contemplated framework may include a view synthesis (stage 1) and a multi-view stereo matching (stage 2). The combination of the two-stage process may provide the advantage of the geometric constraints from stereo matching to improve depth map quality, without the need of additional input data. It is also contemplated that a spherical warping layer may be used to integrate multiple spherical features volumes to one cost volume with uniformly sampled inverse depth for the multi-view spherical stereo matching stage. The two-stage spherical depth estimation system and method may be used in various applications including virtual reality, autonomous driving and robotics.
1. A method for employing a spherical warping layer machine learning algorithm within a spherical stereo matching network, comprising:
receiving one or more equirectangular projection (ERP) images from a sensor;
receiving a feature map from a spherical feature extraction machine learning algorithm; and
transforming an inverse depth hypothesis to a displacement in a spherical-coordinate domain, wherein the displacement is generated using a cosine angle of one or more latitudinal values across the one or more ERP images, one or more baseline values, and a height of the feature map.
2. The method of claim 1 , wherein the inverse depth hypothesis covers a whole depth range.
3. The method of claim 1 , further comprising: mapping one or more pixels from one or more synthesized images to the ERP image.
4. The method of claim 3 , wherein the spherical stereo matching network includes two cascaded stages for disparity prediction in a coarse-to-fine manner, wherein a multi-view stereo matching machine learning algorithm is operable to generate a final spherical depth map.
5. The method of claim 4 , wherein the spherical stereo matching network further includes a spherical feature extraction algorithm operable to pair the one or more synthesized images with the ERP image.
6. The method of claim 5 , wherein the spherical stereo matching network generates one or more feature maps.
7. The method of claim 6 , wherein the one or more feature maps have a resolution of 32 multiplied by one fourth of a width of the ERP image and one fourth of a second height of the ERP image.
8. The method of claim 6 , wherein the spherical stereo matching network further includes a cost volume construction algorithm that aggregates spherical features extracted from the ERP image and the one or more synthesized images.
9. The method of claim 8 , wherein the cost volume construction algorithm generates a cost volume using a depth hypothesis that is uniformly sampled.
10. The method of claim 9 , wherein the one or more feature maps are used by the cost volume construction algorithm to generate the cost volume.
11. The method of claim 10 , wherein the spherical stereo matching network further includes a cost aggregation machine learning algorithm operable to aggregate the cost volume using one or more 3-dimensional convolutional neural networks.
12. The method of claim 11 , wherein the cost aggregation machine learning algorithm is implemented using an hourglass encoding and decoding processes.
13. The method of claim 11 , wherein the spherical stereo matching network further includes a regression algorithm that regresses a disparity value pixel-wise for each specified level.
14. The method of claim 9 , wherein the depth hypothesis is uniformly sampled at a specified level using new intervals.
15. A system for employing a spherical warping layer machine learning algorithm within a spherical stereo matching network, comprising:
a sensor operable to capture one or more equirectangular projection (ERP) images;
a controller operable to:
receive a feature map from a spherical feature extraction machine learning algorithm; and
transform an inverse depth hypothesis to a displacement in a spherical-coordinate domain, wherein the displacement is generated using a cosine angle of one or more latitudinal values across the one or more ERP images, one or more baseline values, and a height of the feature map.
16. The system of claim 15 , wherein the inverse depth hypothesis covers a whole depth range.
17. The system of claim 15 , further comprising: mapping one or more pixels from one or more synthesized images to the ERP image.
18. A method for employing a spherical warping layer machine learning algorithm within a spherical stereo matching network, comprising:
receiving one or more equirectangular projection (ERP) images from a sensor;
receiving a feature map from a spherical feature extraction machine learning algorithm;
transforming a sampled disparity to a displacement in a spherical-coordinate domain, wherein the displacement is generated using a cosine angle of one or more latitudinal values across the one or more ERP images, one or more baseline values, and a height of the feature map; and
configuring a physical system to operate using the displacement in the spherical-coordinate domain.
19. The method of claim 18 , further comprising: transforming an inverse depth hypothesis to the displacement in the spherical coordinate domain, wherein the inverse depth hypothesis covers a whole depth range.
20. The method of claim 18 , further comprising: mapping one or more pixels from one or more synthesized images to the ERP image.