Sparse and/or dense depth estimation from stereoscopic imaging
Implementations are described herein for performing depth estimation in the agricultural domain, including generating synthetic training data. In various implementations, one or more three-dimensional synthetic plants may be generated in in a three-dimensional space, wherein the one or more three-dimensional synthetic plants include homogenous and densely-distributed synthetic plant parts. The plurality of three-dimensional synthetic plants may be projected onto two-dimensional planes from first and second perspectives in the three-dimensional space to form a pair of synthetic stereoscopic images. The first and second synthetic stereoscopic images of the pair may be annotated to create a mapping between the individual synthetic plant parts across the first synthetic stereoscopic images. A feature matching machine learning model may be trained based on the mapping.
1. A method for generating a pair of synthetic stereoscopic training images, the method implemented using one or more processors and comprising:
selecting one or more characteristics for generation of one or more images of three-dimensional synthetic plants based on an environmental consideration of an agricultural area, the one or more characteristics to correspond to data to generate a plant model;
generating the one or more images of the three-dimensional synthetic plants in a three-dimensional space based on the one or more characteristics, wherein the one or more three-dimensional synthetic plants include homogenous and densely distributed synthetic plant parts;
projecting the one or more images of the three-dimensional synthetic plants onto a two-dimensional plane from a first perspective in the three-dimensional space to form a first synthetic stereoscopic image of the pair;
projecting the one or more images of the three-dimensional synthetic plants onto another two-dimensional plane from a second perspective in the three-dimensional space to form a second synthetic stereoscopic image of the pair;
annotating pixels of the first and second synthetic stereoscopic images that represent same individual synthetic plant parts to create a mapping between depictions of the same individual synthetic plant parts across the first and second synthetic stereoscopic images; and
training a feature matching machine learning model based on the mapping.
2. The method of claim 1 , wherein the homogenous and densely distributed synthetic plant parts include individual blades of grass.
3. The method of claim 1 , wherein the homogenous and densely distributed synthetic plant parts include individual leaves of the one or more images of the three-dimensional synthetic plants.
4. The method of claim 1 , wherein the homogenous and densely distributed synthetic plant parts include individual stems or culms of the one or more three-dimensional synthetic plants.
5. The method of claim 1 , wherein the homogenous and densely distributed synthetic plant parts include individual flowers or pedicles of the one or more three-dimensional synthetic plants.
6. The method of claim 1 , wherein the homogenous and densely distributed synthetic plant parts include individual peduncles of the one or more images of the three-dimensional synthetic plants.
7. A method for performing depth estimation, the method implemented using one or more processors and comprising:
comparing one or more spatial aspects of a first bounding box visual annotation of a first candidate plant trait instance depicted in a first stereo image to one or more spatial aspects of a second bounding box visual annotation of a second candidate plant trait instance depicted in a second stereo image;
based on the comparing, calculating an annotation-based similarity score between the first and second candidate plant trait instances across the first and second stereo images;
processing the first and second stereo images based on a feature matching machine learning model to determine a feature-based similarity score between the first and second candidate plant trait instances across the first and second stereo images;
calculating a composite similarity score based on the annotation-based similarity score and the feature-based similarity score;
determining that the first and second candidate plant trait instances are a match for a same plant trait instance across the first and second stereo images, wherein the determination that the first and second candidate plant trait instances are a match for the same plant trait instance across the first and second stereo images is based on the composite similarity score;
determining a disparity between the first and second candidate plant trait instances;
based on the disparity, estimating a depth associated with the same plant trait instance; and
causing output to be provided at a computing device, wherein the output is generated based on the depth associated with the same plant trait instance.
8. The method of claim 7 , wherein determining that the first and second candidate plant trait instances are a match for the same plant trait instance across the first and second stereo images includes performing bipartite graph matching.
9. The method of claim 7 , further including matching one or more additional phenotypic traits of a plant depicted in the first stereo image having the first candidate plant trait instance to one or more additional phenotypic traits of a plant depicted in the second stereo image having the second candidate plant trait instance, wherein the determining that the first and second candidate plant trait instances are a match is further based on the matching.
10. The method of claim 7 , wherein the annotation-based similarity score and the feature-based similarity score are weighted differently when calculating the composite similarity score.
11. The method of claim 7 , wherein the annotation-based similarity score and the feature-based similarity score are used to corroborate and/or refute each other.
12. The method of claim 7 , wherein the feature matching machine learning model includes a convolutional neural network.
13. A system for generating a pair of synthetic stereoscopic training images, the system comprising one or more processors and memory storing instructions that, in response to execution by the one or more processors, cause the one or more processors to:
select one or more characteristics for generation of one or more images of three-dimensional synthetic plants based on an environmental consideration of an agricultural area, the one or more characteristics to correspond to data to generate a plant model;
generate the one or more images of the three-dimensional synthetic plants in a three-dimensional space based on the one or more characteristics, wherein the one or more three-dimensional synthetic plants include homogenous and densely distributed synthetic plant parts;
project the one or more images of the three-dimensional synthetic plants onto a two-dimensional plane from a first perspective in the three-dimensional space to form a first synthetic stereoscopic image of the pair;
project the one or more images of the three-dimensional synthetic plants onto another two-dimensional plane from a second perspective in the three-dimensional space to form a second synthetic stereoscopic image of the pair;
annotate pixels of the first and second synthetic stereoscopic images that represent same individual synthetic plant parts to create a mapping between depictions of the same individual synthetic plant parts across the first and second synthetic stereoscopic images; and
train a feature matching machine learning model based on the mapping.
14. The system of claim 13 , wherein the homogenous and densely distributed synthetic plant parts include individual blades of grass.
15. The system of claim 13 , wherein the homogenous and densely distributed synthetic plant parts include individual leaves of the one or more images of the three-dimensional synthetic plants.
16. The system of claim 13 , wherein the homogenous and densely distributed synthetic plant parts include individual stems or culms of the one or more three-dimensional synthetic plants.
17. The system of claim 13 , wherein the homogenous and densely distributed synthetic plant parts include individual flowers or pedicles of the one or more three-dimensional synthetic plants.
18. The system of claim 13 , wherein the homogenous and densely distributed synthetic plant parts include individual peduncles of the one or more images of the three-dimensional synthetic plants.