Planar region guided 3D geometry estimation from a single image
View Patent ↗Techniques for planar region-guided estimates of 3D geometry of objects depicted in a single 2D image. The techniques estimate regions of an image that are part of planar regions (i.e., flat surfaces) and use those planar region estimates to estimate the 3D geometry of the objects in the image. The planar regions and resulting 3D geometry are estimated using only a single 2D image of the objects. Training data from images of other objects is used to train a CNN with a model that is then used to make planar region estimates using a single 2D image. The planar region estimates, in one example, are based on estimates of planarity (surface plane information) and estimates of edges (depth discontinuities and edges between surface planes) that are estimated using models trained using images of other scenes.
1. A method, performed by a computing device, for enhancing an image based on planar-region-guided estimates of 3D geometry of objects depicted in the image, the method comprising:
determining planarity and edge strength of pixels of the image, wherein the determining of planarity and edge strength of the pixels of the image is based on the image of the objects and is not based on additional images of the objects;
determining whether pixels of the image are within common planar regions based on the determining of planarity and edge strength of the pixels of the image;
determining 3D geometry values of pixels in the common planar regions based on a planar region constraint that requires a relationship between the 3D geometry values of pixels within common planar regions; and
enhancing the image by using the 3D geometry values of the pixels to provide the 3D geometry of the objects in the image.
2. The method of claim 1 , wherein determining the 3D geometry values of the pixels in common planar regions comprises selecting normals of the pixels of the image based on the planar region constraint requiring similar normals of pixels in common planar regions.
3. The method of claim 1 , wherein determining the 3D geometry values of the pixels in common planar regions comprises selecting depths of the pixels of the image based on the planar region constraint requiring that depths of pixels in common planar regions translate to pixel 3D positions consistent with planar expectations.
4. The method of claim 1 , wherein determining the 3D geometry values of the pixels in common planar regions comprises:
selecting normals of the pixels of the image based on the planar region constraint requiring similar normals of pixels in common planar regions; and
selecting depths of the pixels of the image based on the planar region constraint requiring that depths of pixels in common planar regions translate to pixel 3D positions consistent with planar expectations.
5. The method of claim 1 , wherein determining the 3D geometry values of the pixels in common planar regions comprises selecting depths and normals of the pixels of the image based on the planar region constraint requiring that a vector connecting 3D pixel positions of a pair of pixels in a common planar region be perpendicular to a normal of a pixel of the pair of pixels.
6. The method of claim 1 , wherein determining the 3D geometry values of the pixels in common planar regions comprises:
receiving original estimates of normals and depths of the pixels of the image;
selecting normals and depths as the 3D geometry values of the pixels by employing an objective function, wherein employing the objective function comprises penalizing deviations from the original estimates of depths and normals and penalizing planar region irregularities based on deviations from the planar region constraint.
7. The method of claim 6 , wherein employing the objective function comprises assigning a relatively smaller penalty to deviations from less confident original estimates of depths and normals.
8. The method of claim 6 further comprising training deep convolutional neural networks (CNNs) using training data and applying the deep CNNs to the image to identify the original estimates of the normals of the pixels, the original estimates of the depths of the pixels, the planarity of the pixels, and the edge strength of the pixels.
9. The method of claim 1 , wherein determining the 3D geometry values of the pixels in common planar regions comprises applying the planar region constraint based on distance between pixels.
10. The method of claim 1 , wherein determining whether pixels of the image are within common planar regions comprises:
deriving a feature vector of the pixels of the image based on the determining of planarity and edge strength of the pixels of the image, wherein deriving the feature vector comprises determining features of pixels of the image; and
comparing the features of pixels of the image to determine whether pixels of the image are within common planar regions.
11. The method of claim 10 , wherein comparing the features of pixels of the image comprises comparing the features of non-local pixels without constraint to spatial proximity of the non-local pixels in the image.
12. The method of claim 1 , wherein determining whether pixels of the image are within common planar regions comprises computing values representing likelihoods that pixels are within common planar regions.
13. A method, performed by a computing device, for enhancing an image with planar-region-guided estimates of 3D geometry of objects depicted in the image, the method comprising:
determining normals, depths, planarity, and edge strength of pixels of the image;
determining whether pixels of the image are within common planar regions based on the planarity and edge strength of the pixels;
adjusting the normals and depths of the pixels by employing an objective function to minimize planar region irregularities, wherein employing the objective function comprises penalizing adjustments to the normals and depths based on amount of adjustment and penalizing deviations from a planar region constraint that requires a relationship between the normals or depths of pixels within common planar regions; and
enhancing the image by using the normals and depths of the pixels to provide the 3D geometry of the objects in the image.
14. The method of claim 13 , wherein penalizing adjustments to the normals and depths based on amount of adjustment comprises penalizing the adjustments based on confidence in the normals and depths.
15. The method of claim 13 , wherein determining normals, depths, planarity, and edge strength of pixels of the image comprises using deep convolutional neural networks (CNNs).
16. The method of claim 13 , wherein determining whether pixels of the image are within common planar regions comprises:
deriving a feature vector of the pixels of the image based on the determining of planarity and edge strength of the pixels of the image, wherein deriving the feature vector comprises determining features of pixels of the image; and
comparing the features of pixels of the image to determine whether pixels of the image are within common planar regions.
17. The method of claim 16 , wherein determining whether pixels of the image are within common planar regions comprises computing values representing likelihoods that pixels are within common planar regions.
18. A method, performed by a computing device, for enhancing an image with planar-region-guided estimates of 3D geometry of objects depicted in the image, the method comprising:
computing values representing likelihoods that pixels of the image are within common planar regions; and
determining 3D geometry values of pixels based on applying a planar region constraint that requires a relationship between the 3D geometry values of pixels within common planar regions, wherein applying the planar region constraint uses the values representing likelihoods that pixels of the image are within common planar regions; and
enhancing the image by using the 3D geometry values of the pixels to provide the 3D geometry of the objects in the image.
19. The method of claim 18 , wherein determining the 3D geometry values of the pixels in common planar regions comprises:
selecting normals of the pixels of the image based on the planar region constraint requiring similar normals of pixels in common planar regions; and
selecting depths of the pixels of the image based on the planar region constraint requiring that depths of pixels in common planar regions translate to pixel 3D positions consistent with planar expectations.
20. The method of claim 18 , wherein computing the values representing likelihoods that pixels are within common planar regions comprises:
deriving a feature vector of the pixels of the image based on the determining of planarity and edge strength of the pixels of the image, wherein deriving the feature vector comprises determining features of pixels of the image; and
comparing the features of pixels of the image to compute the values representing likelihoods that pixels are within common planar regions.