Techniques for enhanced image capture using a computer-vision network
Disclosed are techniques for enhancing two-dimensional (2D) image capture of subjects (e.g., a physical structure, such as a residential building) to maximize the feature correspondences available for three-dimensional (3D) model reconstruction. More specifically, disclosed is a computer-vision network configured to provide viewfinder interfaces and analyses to guide the improved capture of an intended subject for specified purposes. Additionally, the computer-vision network can be configured to generate a metric representing a quality of feature correspondences between images of a complete set of images used for reconstructing a 3D model of a physical structure. The computer-vision network can also be configured to generate feedback at or before image capture time to guide improvements to the quality of feature correspondences between a pair of images.
1. A computer-implemented method, comprising:
capturing a set of pixels representing a scene visible to an image capturing device including a display, the set of pixels including a plurality of border pixels, and each border pixel of the plurality of border pixels being located at or within a defined range of a boundary of the set of pixels;
detecting a physical structure depicted within the set of pixels, the physical structure being represented by a subset of the set of pixels;
generating a segmentation mask associated with the physical structure depicted within the set of pixels, the segmentation mask including one or more segmentation pixels, wherein the segmentation mask comprises an irregular shape that conforms to contours of the subset of the set of pixels;
determining a pixel value for each border pixel of the plurality of border pixels;
generating an indicator based on the pixel value of one or more border pixels of the plurality of border pixels by:
determining a number of the one or more border pixels with pixel values indicating overlap with the one or more segmentation pixels; and
generating the indicator based on whether the number of the one or more border pixels with pixel values indicating overlap with the one or more segmentation pixels exceeds a threshold percentage of the plurality of border pixels; and
presenting the indicator, the indicator representing an instruction for framing the physical structure within the display.
2. The computer-implemented method of claim 1 , wherein determining the pixel value for each border pixel further comprises:
detecting that the one or more border pixels of the plurality of border pixels includes a segmentation pixel of the one or more segmentation pixels, and wherein the plurality of border pixels includes:
one or more left edge border pixels located at a left edge of the set of pixels;
one or more or more top edge border pixels located at a top edge of the set of pixels;
one or more right edge border pixels located at a right edge of the set of pixels; and
one or more bottom edge border pixels located at a bottom edge of the set of pixels.
3. The computer-implemented method of claim 2 , wherein:
when a left edge border pixel of the one or more left edge border pixels includes a segmentation pixel, the instruction represented by the indicator instructs a user viewing the display to move the image capturing device in a leftward direction;
when a top edge border pixel of the one or more top edge border pixels includes a segmentation pixel, the instruction represented by the indicator instructs the user viewing the display to move the image capturing device in an upward direction;
when a right edge border pixel of the one or more right edge border pixels includes a segmentation pixel, the instruction represented by the indicator instructs the user viewing the display to move the image capturing device in a rightward direction; and
when a bottom edge border pixel of the one or more bottom edge border pixels includes a segmentation pixel, the instruction represented by the indicator instructs the user viewing the display to move the image capturing device in a downward direction.
4. The computer-implemented method of claim 2 , wherein:
when each of a left edge border pixel, a top edge border pixel, a right edge border pixel, and a bottom edge border pixel includes a segmentation pixel, the instruction represented by the indicator instructs a user viewing the display to move backward.
5. The computer-implemented method of claim 2 , wherein when none of the one or more left edge border pixels, the one or more top edge border pixels, the one or more right edge border pixels, and the one or more bottom edge border pixels includes a segmentation pixel, the instruction represented by the indicator instructs a user viewing the display to zoom in to frame the physical structure.
6. The computer-implemented method of claim 1 , wherein the plurality of border pixels comprises a border having a pixel width determined by a width of the set of pixels.
7. The computer-implemented method of claim 1 , wherein presenting the indicator comprises:
displaying the indicator on the display of the image capturing device; or
audibly presenting the indicator to a user operating the image capturing device.
8. The computer-implemented method of claim 1 , wherein the border pixels comprise fields storing image information and a field to store the pixel value indicating whether a border pixel intersects with the segmentation mask.
9. A computer-program product tangibly embodied in a non-transitory machine-readable storage medium, including instructions configured to cause a data processing apparatus to perform operation including:
capturing a set of pixels representing a scene visible to an image capturing device including a display, the set of pixels including a plurality of border pixels, and each border pixel of the plurality of border pixels being located at or within a defined range of a boundary of the set of pixels;
detecting a physical structure depicted within the set of pixels, the physical structure being represented by a subset of the set of pixels;
generating a segmentation mask associated with the physical structure depicted within the set of pixels, the segmentation mask including one or more segmentation pixels, wherein the segmentation mask comprises an irregular shape that conforms to contours of the subset of the set of pixels;
determining a pixel value for each border pixel of the plurality of border pixels;
generating an indicator based on the pixel value of one or more border pixels of the plurality of border pixels by:
determining a number of the one or more border pixels with pixel values indicating overlap with the one or more segmentation pixels; and
generating the indicator based on whether the number of the one or more border pixels with pixel values indicating overlap with the one or more segmentation pixels exceeds a threshold percentage of the plurality of border pixels; and
presenting the indicator, the indicator representing an instruction for framing the physical structure within the display.
10. The computer-program product of claim 9 , wherein the threshold percentage is a function of a related pixel dimension of the segmentation mask.
11. The computer-program product of claim 9 , wherein the threshold percentage comprises a predetermined number of consecutive border pixels in the plurality of border pixels that overlap with the one or more segmentation pixels.
12. The computer-program product of claim 9 , wherein the defined range of the boundary of the set of pixels comprises between 2 and 10 pixels from the boundary of the set of pixels, such that the plurality of border pixels have a width of between 2 and 10 pixels around the boundary of the set of pixels.
13. The computer-program product of claim 9 , wherein the segmentation mask is generated by a classifier trained to identify physical structures in sets of pixels.
14. The computer-program product of claim 9 , further comprising:
generating a bounding box around the segmentation mask, such that all of the one or more segmentation pixels fit inside of the bounding box, and the bounding box includes a buffer region such that the bounding box does not tangentially touch any of the one or more segmentation pixels; and
presenting the bounding box and the set of pixels with the indicator.
15. The computer-program product of claim 9 , further comprising:
denoising the segmentation mask by smoothing a boundary of the segmentation mask.
16. The computer-program product of claim 15 , wherein
denoising the segmentation mask comprises combining a plurality of segmentation masks from a plurality of frames of the scene that include the physical structure.
17. The computer-program product of claim 16 , wherein combining the plurality of segmentation masks from the plurality of frames comprises pixel voting using pixels from the plurality of frames.
18. The computer-program product of claim 17 , wherein the pixel voting is weighted based on a motion of the image capturing device when capturing each of the plurality of frames.