Panoramic camera systems
A camera system captures images from a set of cameras to generate binocular panoramic views of an environment. The cameras are oriented in the camera system to maximize the minimum number of cameras viewing a set of randomized test points. To calibrate the system, matching features between images are identified and used to estimate three-dimensional points external to the camera system. Calibration parameters are modified to improve the three-dimensional point estimates. When images are captured, a pipeline generates a depth map for each camera using reprojected views from adjacent cameras and an image pyramid that includes individual pixel depth refinement and filtering between levels of the pyramid. The images may be used generate views of the environment from different perspectives (relative to the image capture location) by generating depth surfaces corresponding to the depth maps and blending the depth surfaces.
1. A method comprising:
receiving a reference image comprising a plurality of pixels, a depth map for the reference image, and a set of overlap images, the depth map comprising a depth estimate for each pixel of the plurality of pixels of the reference image;
generating a set of features for the reference image; and
refining the depth map for one or more pixels of the plurality of pixels of the reference image by:
generating a first set of proposals for the depth estimate of a first pixel of the plurality of pixels, the set of proposals comprising the current depth estimate for the first pixel and the depth estimate for a neighboring pixel to the first pixel;
calculating, based on the set of features, a cost for each proposal of the first set of proposals using a cost function;
selecting a proposal from the first set of proposals based on the calculated cost of each proposal of the first set of proposals;
determining a first refined depth estimate for the first pixel by performing a first step of gradient descent on the selected proposal of the first set of proposals;
setting the first refined depth estimate as the current depth estimate for the first pixel;
responsive to updating the current depth estimate for the first pixel, generating a second set of proposals for the depth estimate of a second pixel of the plurality of pixels, the second pixel neighboring the first pixel and the set of proposals comprising the current depth estimate for the second pixel and the current depth estimate for the first pixel;
calculating, based on the generated features, a cost for each proposal of the second set of proposals using the cost function;
selecting, from the second set of proposals, the current depth estimate for the first pixel based on the calculated cost of each proposal of the second set of proposals, wherein the current depth estimate for the first pixel is selected when the current depth estimate for the first pixel is associated with the lowest cost of the set of proposals; and
determining a second refined depth estimate for the second pixel by performing a second step of gradient descent on the current depth estimate for the first pixel based on the first step of gradient descent; and
setting the second refined depth estimate as the current depth estimate for the second pixel.
2. The method of claim 1 , wherein the set of features for the reference image comprises a set of reprojected overlap images generated by reprojecting each overlap image of the set of overlap images to the viewpoint of the reference image with an estimate of infinity depth.
3. The method of claim 2 , wherein the cost function comprises one or more depth accuracy terms, wherein a depth accuracy term compares the current pixel of the reference image to a corresponding pixel in a reprojected overlap image of the set of reprojected overlap images determined based on the current depth estimate.
4. The method of claim 2 , wherein the set of features further comprises intensity and intensity gradient features for the reference image and the set of reprojected overlap images.
5. The method of claim 1 , wherein the cost function comprises a regularization term biasing the cost function to prefer larger depth estimates.
6. The method of claim 5 , wherein regularization term is calculated based on the variance of the depth map at the current pixel.
7. The method of claim 1 , wherein the first step of gradient descent and the second step of gradient descent are performed based on the gradient of the cost function.
8. The method of claim 1 , further comprising determining a pixel order progressing sequentially through each of the plurality of pixels of the reference image and wherein refining the depth map for each pixel of the plurality of pixels of the reference image comprises sequentially selecting pixels for refinement following the determined pixel order, wherein the first pixel occurs previously in the order than the second pixel.
9. The method of claim 8 , wherein refining the depth map for each pixel of the plurality of pixels of the reference image further comprises sequentially selecting pixels for refinement following the reverse of the determined pixel order.
10. A non-transitory computer readable storage medium comprising instructions which, when executed by a processor, cause the processor to perform the steps of:
receiving a reference image comprising a plurality of pixels, a depth map for the reference image, and a set of overlap images, the depth map comprising a depth estimate for each pixel of the plurality of pixels of the reference image;
generating a set of features for the reference image; and
refining the depth map for one or more pixels of the plurality of pixels of the reference image by:
generating a first set of proposals for the depth estimate of a first pixel of the plurality of pixels, the set of proposals comprising the current depth estimate for the first pixel and the depth estimate for a neighboring pixel to the first pixel;
calculating, based on the set of features, a cost for each proposal of the first set of proposals using a cost function;
selecting a proposal from the first set of proposals based on the calculated cost of each proposal of the first set of proposals;
determining a first refined depth estimate for the first pixel by performing a first step of gradient descent on the selected proposal of the first set of proposals;
setting the first refined depth estimate as the current depth estimate for the first pixel;
responsive to updating the current depth estimate for the first pixel, generating a second set of proposals for the depth estimate of a second pixel of the plurality of pixels, the second pixel neighboring the first pixel and the set of proposals comprising the current depth estimate for the second pixel and the current depth estimate for the first pixel;
calculating, based on the generated features, a cost for each proposal of the second set of proposals using the cost function;
selecting, from the second set of proposals, the current depth estimate for the first pixel based on the calculated cost of each proposal of the second set of proposals, wherein the current depth estimate for the first pixel is selected when the current depth estimate for the first pixel is associated with the lowest cost of the set of proposals; and
determining a second refined depth estimate for the second pixel by performing a second step of gradient descent on the current depth estimate for the first pixel based on the first step of gradient descent; and
setting the second refined depth estimate as the current depth estimate for the second pixel.
11. The non-transitory computer readable storage medium of claim 10 , wherein the set of features for the reference image comprises a set of reprojected overlap images generated by reprojecting each overlap image of the set of overlap images to the viewpoint of the reference image with an estimate of infinity depth.
12. The non-transitory computer readable storage medium of claim 11 , wherein the cost function comprises one or more depth accuracy terms, wherein a depth accuracy term compares the current pixel of the reference image to a corresponding pixel in a reprojected overlap image of the set of reprojected overlap images determined based on the current depth estimate.
13. The non-transitory computer readable storage medium of claim 11 , wherein the set of features further comprises intensity and intensity gradient features for the reference image and the set of reprojected overlap images.
14. The non-transitory computer readable storage medium of claim 10 , wherein the cost function comprises a regularization term biasing the cost function to prefer larger depth estimates.
15. The non-transitory computer readable storage medium of claim 14 , wherein regularization term is calculated based on the variance of the depth map at the current pixel.
16. The non-transitory computer readable storage medium of claim 10 , wherein the first step of gradient descent step and the second step of gradient descent are performed based on the gradient of the cost function.
17. The non-transitory computer readable storage medium of claim 10 , further comprising determining a pixel order progressing sequentially through each of the plurality of pixels of the reference image and wherein refining the depth map for each pixel of the plurality of pixels of the reference image comprises sequentially selecting pixels for refinement following the determined pixel order, wherein the first occurs previously in the order than the second pixel.
18. The non-transitory computer readable storage medium of claim 17 , wherein refining the depth map for each pixel of the plurality of pixels of the reference image further comprises sequentially selecting pixels for refinement following the reverse of the determined pixel order.
19. A system comprising:
an image processing system configured to:
receive a reference image comprising a plurality of pixels, a depth map for the reference image, and a set of overlap images, the depth map comprising a depth estimate for each pixel of the plurality of pixels of the reference image;
generate a set of features for the reference image; and
refine the depth map for one or more pixels of the plurality of pixels of the reference image by:
generating a first set of proposals for the depth estimate of a first pixel of the plurality of pixels, the set of proposals comprising the current depth estimate for the first pixel and the depth estimate for a neighboring pixel to the first pixel;
calculating, based on the set of features, a cost for each proposal of the first set of proposals using a cost function;
selecting a proposal from the first set of proposals based on the calculated cost of each proposal of the first set of proposals;
determining a first refined depth estimate for the first pixel by performing a first step of gradient descent on the selected proposal of the first set of proposals;
setting the first refined depth estimate as the current depth estimate for the first pixel;
responsive to updating the current depth estimate for the first pixel, generating a second set of proposals for the depth estimate of a second pixel of the plurality of pixels, the second pixel neighboring the first pixel and the set of proposals comprising the current depth estimate for the second pixel and the current depth estimate for the first pixel;
calculating, based on the generated features, a cost for each proposal of the second set of proposals using the cost function;
selecting, from the second set of proposals, the current depth estimate for the first pixel based on the calculated cost of each proposal of the second set of proposals, wherein the current depth estimate for the first pixel is selected when the current depth estimate for the first pixel is associated with the lowest cost of the set of proposals; and
determining a second refined depth estimate for the second pixel by performing a second step of gradient descent on the current depth estimate for the first pixel based on the first step of gradient descent; and
setting the second refined depth estimate as the current depth estimate for the second pixel.
20. The system of claim 19 , wherein the set of features for the reference image comprises a set of reprojected overlap images generated by reprojecting each overlap image of the set of overlap images to the viewpoint of the reference image with an estimate of infinity depth.