Three-dimensional modeling with two dimensional data
Implementations are described herein for three-dimensional (“3D”) modeling of objects that target specific features of interest of the objects, and ignore other features of less interest. In various implementations, a plurality of two-dimensional (“2D”) images may be received from a 2D vision sensor. The plurality of 2D images may capture an object having multiple classes of features. Data corresponding to a first set of the multiple classes of features may be filtered from the plurality of 2D images to generate a plurality of filtered 2D images in which a second set of features of the multiple classes of features is captured. 2D-3D processing, such as structure from motion (“SFM”) processing, may be performed on the 2D filtered images to generate a 3D representation of the object that includes the second set of one or more features.
1. A method implemented using one or more processors, comprising:
receiving, via a computing device operated by a user, input that identifies the user's viewing perspective while the user operates the computing device to navigate through a virtual rendering of an area of interest;
based on the user's viewing perspective, retrieving from a database a plurality of two-dimensional (“2D”) images captured by a 2D vision sensor, wherein the plurality of 2D images capture a plant in the area of interest having multiple classes of plant features;
filtering data corresponding to a first set of one or more of the multiple classes of plant features from the plurality of 2D images to generate a plurality of filtered 2D images, wherein the plurality of filtered 2D images capture a second set of one or more of the multiple classes of plant features;
performing structure from motion (“SFM”) processing on the plurality of 2D filtered images to generate a three-dimensional (“3D”) representation of the plant, wherein the 3D representation of the plant includes the second set of one or more of the multiple classes of plant features; and
providing, at the computing device, output that conveys one or more aspects of the 3D representation of the plant.
2. The method of claim 1 , wherein the 3D representation of the plant excludes the first set of the one or more of the multiple classes of plant features.
3. The method of claim 1 , further comprising applying the plurality of 2D images as input across a trained machine learning model to generate output data, wherein the output data semantically classifies pixels of the plurality of 2D images into the multiple classes of plant features.
4. The method of claim 3 , wherein the filtering includes filtering pixels classified into one or more of the first set of one or more classes of plant features from the plurality of 2D images.
5. The method of claim 3 , wherein the trained machine learning model comprises a convolutional neural network.
6. The method of claim 1 , wherein the filtering includes locating one or more bounding boxes around plants identified as members of one or more of the second set of multiple classes of plant features.
7. The method of claim 1 , wherein the multiple classes of plant features include two or more of leaf, fruit, branch, soil, and stem, and wherein the one or more aspects of the 3D representation of the plant include one or more of:
a statistic about fruit of the plant;
a statistic about leaves of the plant;
a statistic about branches of the plant;
a statistic about buds of the plant;
a statistic about flowers of the plant; or
a statistic about panicles of the plant.
8. The method of claim 1 , wherein the computing device comprises a virtual reality (“VR”) or augmented reality (“AR”) headset, and the input comprises eye tracking data or sensor data that conveys a direction the user is facing.
9. The method of claim 1 , wherein the 3D representation of the plant comprises a first 3D representation of the plant, and the method further comprises:
filtering data corresponding to a third set of one or more of the multiple classes of plant features from the plurality of 2D images to generate a second plurality of filtered 2D images, wherein the second plurality of filtered 2D images capture a fourth set of one or more plant features of the multiple classes of plant features; and
performing SFM processing on the second plurality of filtered images to generate a second 3D representation of the plant, wherein the second 3D representation of the plant includes the fourth set of one or more plant features;
wherein the output comprises a graphical user interface in which the first and second 3D representations of the plant are selectably renderable as layers.
10. A method implemented using one or more processors, comprising:
receiving, via a computing device operated by a user, input that identifies the user's viewing perspective while the user operates the computing device to navigate through a virtual rendering of an area of interest;
based on the user's viewing perspective, retrieving from a database receiving a plurality of two-dimensional (“2D”) images captured by a 2D vision sensor;
applying the plurality of 2D images as input across a trained machine learning model to generate output data, wherein the output data semantically segments the plurality of 2D images into a plurality of semantic classes;
performing structure from motion (“SFM”) processing on one or more selected semantic classes of the plurality of 2D images to generate a three-dimensional (“3D”) representation of an object, wherein the 3D representation of the object excludes one or more unselected semantic classes of the plurality of 2D images; and
providing, at the client computing device, output that conveys one or more aspects of the 3D representation of the object.
11. The method of claim 10 , wherein the trained machine learning model comprises a convolutional neural network.
12. The method of claim 10 , wherein the object comprises a plant, the plurality of semantic classes include two or more of leaf, fruit, branch, soil, stem, flower, bud, and panicle.
13. The method of claim 10 , wherein the computing device comprises a virtual reality (“VR”) or augmented reality (“AR”) headset, and the input comprises eye tracking data or sensor data that conveys a direction the user is facing.
14. At least one non-transitory computer-readable medium comprising instructions that, in response to execution of the instructions by one or more processors, cause the one or more processors to perform the following operations:
receiving, via a computing device operated by a user, input that identifies the user's viewing perspective while the user operates the computing device to navigate through a virtual rendering of an area of interest;
based on the user's viewing perspective, retrieving from a database a plurality of two-dimensional (“2D”) images captured by a 2D vision sensor, wherein the plurality of 2D images capture a plant having multiple classes of plant features;
filtering data corresponding to a first set of one or more of the multiple classes of plant features from the plurality of 2D images to generate a plurality of filtered 2D images, wherein the plurality of filtered 2D images capture a second set of one or more plant features of the multiple classes of plant features;
performing two-dimensional-to-three dimensional (“2D-to-3D”) processing on the plurality of 2D filtered images to generate a 3D representation of the plant, wherein the 3D representation of the plant includes the second set of one or more plant features; and
providing, at the computing device, output that conveys one or more aspects of the 3D representation of the plant.
15. The at least one non-transitory computer-readable medium of claim 14 , wherein the 3D representation of the plant excludes the first set of one or more of the multiple classes of plant features.
16. The at least one non-transitory computer-readable medium of claim 14 , further comprising instructions for applying the plurality of 2D images as input across a trained machine learning model to generate output data that semantically classifies pixels of the plurality of 2D images into the multiple classes of plant features.
17. The at least one non-transitory computer-readable medium of claim 16 , wherein the filtering includes filtering pixels classified into one or more of the first set of one or more classes of plant features from the plurality of 2D images.
18. The at least one non-transitory computer-readable medium of claim 16 , wherein the trained machine learning model comprises a convolutional neural network.
19. The at least one non-transitory computer-readable medium of claim 14 , wherein the multiple classes of plant features include two or more of leaf, fruit, branch, soil, and stem.
20. The at least one non-transitory computer-readable medium of claim 19 , wherein the one or more aspects of the 3D representation of the plant include one or more of:
a statistic about fruit of the plant;
a statistic about leaves of the plant;
a statistic about branches of the plant;
a statistic about buds of the plant;
a statistic about flowers of the plant; and
a statistic about panicles of the plant.