Plant feature detection using captured images
Described are methods for identifying the in-field positions of plant features on a plant by plant basis. These positions are determined based on images captured as a vehicle (e.g., tractor, sprayer, etc.) including one or more cameras travels through the field along a row of crops. The in-field positions of the plant features are useful for a variety of purposes including, for example, generating three-dimensional data models of plants growing in the field, assessing plant growth and phenotypic features, determining what kinds of treatments to apply including both where to apply the treatments and how much, determining whether to remove weeds or other undesirable plants, and so on.
1. A method for treating a plant comprising:
receiving a sequence of stereo image pairs captured as a device passes along a row of plants in a field, each image of each stereo image pair of the sequence of stereo image pairs representative of at least a portion of one or more of the plants in the row;
for each stereo image pair of the sequence of stereo image pairs,
generating a probability heatmap corresponding to an image from the stereo image pair and comprising probabilities that pixels within the image contain an occurrence of one or more plant features;
generating a depth map corresponding to the stereo image pair and comprising points in a three-dimensional space, each point in the three-dimensional space corresponding to a pixel in each image of the stereo image pair;
combining the probability heat map and the depth map to generate a combined map representing plants in the row in the three-dimensional space, wherein the combined map comprises a plurality of point clusters representing each plant in the row and describing a probability of an occurrence of one or more plant features in each plant in the row; and
identifying the occurrence of one or more plant features based on the plurality of point clusters in the combined map.
2. The method of claim 1 , further comprising:
receiving the sequence of stereo image pairs from a camera coupled to a farming machine as the farming machine passes along the row of plants in the field; and
correcting each image of the sequence of stereo image pairs to account for at least one optical effect in the images.
3. The method of claim 1 , further comprising:
calibrating the sequence of stereo images, the calibration comprising:
accessing an initial projection used to capture the sequence of stereo images;
generating a desired projection of the sequence of stereo images; and
mapping pixels within stereo images of the sequence from the initial projection to the desired projection.
4. The method of claim 1 , wherein generating the probability heat map further comprises:
inputting the image of the stereo image pair into a machine learned classifier trained using test images of crops including pixels of one or more plant features;
generating, by the machine learned classifier, a heat map matrix of probabilities, wherein individual pixels in the image are associated with individual probabilities in the heat map matrix; and
generating the probability heat map corresponding to the image from the stereo image pair based on the heat map matrix of probabilities.
5. The method of claim 4 , further comprising:
clustering individual pixels in the image within a threshold distance from one another into a pixel group of a set of pixel groups; and
converting each pixel group of the set of pixel groups into a voxel heat map, wherein each voxel of the voxel heat map incorporates image data of the pixels in the pixel group used to generate the voxel.
6. The method of claim 1 , wherein generating a plurality of the depth map comprises:
identifying, from the stereo image pair, a left image and a right image;
determining a depth value for each pixel in the left and right images; and
generating, for the stereo image pair, a depth map comprising a plurality of points, each point of the plurality of points assigned a depth value determined for a pixel in the left or right images corresponding to the point.
7. The method of claim 1 , wherein generating the depth map comprises:
identifying, from the stereo image pair, a left image and a right image;
identifying pixels in each of the left image and right image; and
generating a point cloud comprising a plurality of points, with each point in the plurality of points corresponding to identified pixels in the left and right images.
8. The method of claim 1 , wherein combining the probability heat map and the depth map to generate a combined map comprises:
determining a reference frame for the three-dimensional space by combining the depth map with a two-dimensional map of the row of plants in the field; and
assigning probabilities from the probability heat map to points in the combined map according to the reference frame.
9. The method of claim 1 , further comprising:
aggregating combined maps for a plurality of stereo image pairs of the sequence of stereo image pairs to generate a global map representing plants in the field in the three-dimensional space.
10. The method of claim 1 , wherein combining the probability heat map and depth map to generate a combined map comprises:
grouping points in the depth map within a threshold proximity in the three-dimensional space;
from the grouped points, clustering points having probabilities for representing a particular plant feature above a threshold probability; and
generating the combined map based on the clustered points.
11. The method of claim 1 , wherein identifying the occurrence of one or more plant features based on the plurality of point clusters in the combined map comprises:
determining a location of a current set of point clusters in the combined map not identifying the occurrence of the one or more plant features;
comparing the current set of point clusters to a previous set of point clusters at the location in a previous combined map, the set of previous point clusters identifying the occurrence of the one or more plant features; and
responsive to the comparison, determining the set of current point clusters identifies the occurrence of the one or more plant features at the location.
12. The method of claim 1 , further comprising:
receiving the sequence of stereo image pairs from a plurality of cameras coupled to a farming machine as the farming machine passes along the row of plants in the field.
13. A plant treatment system comprising:
at least one camera configured capture a sequence of stereo image pairs as the camera passes along a row of plants in a field, each image of each stereo pair of the sequence of stereo image pairs representative of capturing at least a portion of one or more of the plants in the row; and
an image capture system comprising computer program instructions that when executed by a computer processor cause the processor to:
receive the sequence of stereo image pairs from the at least one camera as the camera captured as a device passes along the row of plants in the field;
for each stereo image pair of the sequence of stereo image pairs, generate a probability heatmap corresponding to an image from the stereo image pair and comprising probabilities that pixels within the image contain an occurrence of one or more plant features;
generate a depth map corresponding to the stereo image pair and comprising points in a three-dimensional space, each point in the three-dimensional space corresponding to a pixel in each image of the stereo image pair;
combining the probability heat map c and the depth map to generate a combined map representing plants in the row in the three-dimensional space, wherein the combined map comprises a plurality of point clusters representing each plant in the row and describing a probability of an occurrence of one or more plant features in each plant in the row; and
identifying the occurrence of one or more plant features based on the plurality of point clusters in the combined map.
14. The system of claim 13 , further comprising instructions that cause the processor to:
receive the sequence of stereo image pairs from a camera coupled to a farming machine as the farming machine passes along the row of plants in the field; and
correct each image of the sequence of stereo image pairs to account for at least one optical effect in the images.
15. The system of claim 13 , further comprising instructions that cause the processor to:
calibrate the sequence of stereo images, the calibration comprising:
accessing an initial projection used to capture the sequence of stereo images;
generating a desired projection of the sequence of stereo images; and
mapping pixels within stereo images of the sequence from the initial projection to the desired projection.
16. The system of claim 13 , wherein the instructions for generating the plurality of probability heat map further comprises instructions that cause the processor to:
input the image of the stereo image pair into a machine learned classifier trained using test images of crops including pixels of one or more plant features;
generate, by the machine learned classifier, a heat map matrix of probabilities, wherein individual pixels in the image are associated with individual probabilities in the heat map matrix; and
generate the probability heat map corresponding to the image from the stereo image pair based on the heat map matrix of probabilities.
17. The system of claim 16 , further comprising instructions that cause the processor to:
cluster individual pixels in the image within a threshold distance from one another into a pixel group of a set of pixel groups; and
convert each pixel group of the set of pixel groups into a voxel heat map, wherein each voxel of the voxel heat map incorporates image data of the pixels in the pixel group used to generate the voxel.
18. The system of claim 13 , wherein the instructions for generating the depth heat map further cause the processor to:
identify, from the stereo image pair, a left image and a right image;
determine a depth value for each pixel in the left and right images; and
generate, for the stereo image pair, a depth map comprising a plurality of points,. each point of the plurality of points assigned a depth value determined for a pixel in the left or right images corresponding to the point.
19. The system of claim 13 , wherein the instructions for generating the depth map further cause the processor to:
identify, from the stereo image pair, a left image and a right image;
identify pixels in each of the left image and right image; and
generate a point cloud comprising points in the point cloud corresponding to the identified pixels in the left and right images.
20. The system of claim 13 , wherein the instructions for combining the probability heat map and the depth map to generate a combined map further cause the processor to:
determine a reference frame for the three-dimensional space by combining the depth map with a two-dimensional map of the row of plants in the field; and
assign probabilities from the probability heat map to points in the combined map according to the reference frame.
21. The system of claim 13 , further comprising instructions that cause the processor to:
aggregate combined maps for a plurality of stereo image pairs of the sequence of stereo image pairs to generate a global map representing plants in the field in the three-dimensional space.
22. The system of claim 13 , wherein the instructions for combining the probability heat map and the depth map to generate a combined map further cause the processor to:
grouping points in the depth map within a threshold proximity in the three-dimensional space;
from the grouped points, clustering points having probabilities for representing a particular plant feature above a threshold probability; and
generate the combined map based on the clustered points.
23. The system of claim 13 , wherein the instructions for identifying the occurrence of one or more plant features based on the plurality of point clusters further causes the processor to:
determine a location of a current set of point clusters in the combined map not identifying the occurrence of the one or more plant features;
compare the current set of point clusters to a previous set of point clusters at the location in a previous combined map, the set of previous point clusters identifying the occurrence of the one or more plant features; and
responsive to the comparison, determine the set of current point clusters identifies the occurrence of the one or more plant features at the location.
24. The system of claim 13 further comprising:
a plurality of cameras configured capture a sequence of stereo image pairs as the camera passes along a row of plants in a field, each camera of the plurality configured to couple to a farming machine as the farming machine passes along the row of plants in the field.