Plant feature detection using captured images
Described are methods for identifying the in-field positions of plant features on a plant by plant basis. These positions are determined based on images captured as a vehicle (e.g., tractor, sprayer, etc.) including one or more cameras travels through the field along a row of crops. The in-field positions of the plant features are useful for a variety of purposes including, for example, generating three-dimensional data models of plants growing in the field, assessing plant growth and phenotypic features, determining what kinds of treatments to apply including both where to apply the treatments and how much, determining whether to remove weeds or other undesirable plants, and so on.
1. A method for treating a plant with a farming machine comprising:
accessing a probability heatmap comprising a plurality of probability values, the probability heatmap corresponding to an image of a plurality of plants in a row, and wherein each probability value of the plurality of probability values:
corresponds to a pixel of a plurality of pixels in the image,
corresponds to a position of a plurality of positions in the row represented by the corresponding pixel in the image, and
represents a likelihood of a particular plant feature occurring at the position represented by the corresponding pixel in the image;
identifying a first set of probability values in the probability heatmap representing a first plant of the plurality of plants in the row, the first plant identified based on:
a first position of the plurality of positions in the row represented by the first set of probability values, and
a first likelihood represented by the first set of probability values that the particular plant feature for the first plant occurs at the first position;
accessing a depth map comprising a plurality of points in a three-dimensional space, the depth map corresponding to the image of the plurality of plants in the row, and wherein:
each point in the depth map is mapped to pixels in the image, and
one or more points in the depth map corresponds to one or more positions of the plurality of positions in the row;
identifying, in the depth map;
a first cluster of points of the plurality of points in the three-dimensional space representing the first plant at the first position in the row, and
a second cluster of points of the plurality of points in the three-dimensional space representing a second plant at a second position of the plurality of positions in the row, the second position behind the first position relative to the farming machine;
determining the first plant obstructs the second plant in response to:
identifying that the first set of probability values represents the first plant, and
identifying that the second cluster of points represents the second plant positioned behind the first plant; and
actuating a treatment mechanism to treat the second plant by dispensing a treatment fluid at the second position in the row.
2. The method of claim 1 , further comprising:
mapping the first set of probability values in the probability heatmap to the first cluster of points in the three-dimensional space;
determining that the first cluster of points in the three-dimensional space does not represent the first plant based on the position of the first cluster of points in the three-dimensional space; and
responsive to determining that the first cluster of points in the three-dimensional space does not represent the first plant, updating the first set of probability values in the probability heatmap to indicate that the first plant is not is present at the first position in the row.
3. The method of claim 1 , further comprising:
identifying a subset of points of the second cluster of points in the three-dimensional space representing a particular plant feature for the second plant.
4. The method of claim 3 , further comprising:
updating the plurality of probability values of the probability heatmap with one or more additional probability values that indicate that the particular plant feature for the second plant occurs at the second position in the row.
5. The method of claim 3 , further comprising:
identifying, by the depth map, pixels in the image mapped by to the subset of points in the second cluster of points in the three-dimensional space as the particular feature for the second plant.
6. The method of claim 3 , further comprising:
removing the second cluster of points from the three-dimensional space;
responsive to removing the second cluster of points from the three-dimensional space, identifying a set of probability values of the plurality of probability values corresponding to the removed second cluster of points; and
updating the identified set of probability values to indicate that the second plant is not present at positions in the row corresponding to the identified set of probability values.
7. The method of claim 1 , further comprising:
generating a line representing the row in in the three-dimensional space of the depth map;
for each plant of the plurality of plants in the row, identifying a cluster of points in the three-dimensional space corresponding to the plant;
determining a distance between the identified cluster of points in the three-dimensional space and the generated line using a center of mass of the identified cluster of points; and
responsive to determining that the determined distance exceeds a threshold, removing the identified cluster of points from the three-dimensional space.
8. The method of claim 1 , wherein identifying the second cluster of points in the three-dimensional space as the second plant at the second position in the row comprises:
for each point of the second cluster of points in the three-dimensional space, comparing a position of the point in the three-dimensional space to positions of points in the first cluster of points in the three-dimensional space; and
determining that the second cluster of points in the three-dimensional space identifies the second plant based on the comparison.
9. The method of claim 1 , further comprising:
determining, for each pixel in the image, a depth value based on the point of the plurality of points in the three-dimensional space mapped to the pixel by the depth map; and
generating a point cloud representation of the image, wherein each point of the point cloud corresponds to a pixel in the image and is assigned the depth value determined for the corresponding pixel.
10. The method of claim 1 , wherein the farming machine does not treat the identified first plant.
11. A non-transitory computer readable storage medium comprising computer program instructions that, when executed by a computer processor cause the processor to:
access a probability heatmap comprising a plurality of probability values, the probability heatmap corresponding to an image of a plurality of plants in a row, and wherein each probability value of the plurality of probability values:
corresponds to a pixel of a plurality of pixels in the image,
corresponds to a position of a plurality of positions in the row represented by the corresponding pixel in the image, and
represents a likelihood of a particular plant feature occurring at the position represented by the corresponding pixel in the image;
identify a first set of probability values in the probability heatmap representing a first plant of the plurality of plants in the row, the first plant identified based on:
a first position of the plurality of positions in the row represented by the first set of probability values, and
a first likelihood represented by the first set of probability values that the particular plant feature for the first plant occurs at the first position;
access a depth map comprising a plurality of points in a three-dimensional space, the depth map corresponding to the image of the plurality of plants in the row, and wherein:
each point in the depth map is mapped to pixels in the image, and
one or more points in the depth map corresponds to one or more positions of the plurality of positions in the row;
identify, in the depth map;
a first cluster of points of the plurality of points in the three-dimensional space representing the first plant at the first position in the row, and
a second cluster of points of the plurality of points in the three-dimensional space representing a second plant at a second position of the plurality of positions in the row, the second position behind the first position relative to the farming machine;
determine the first plant obstructs the second plant in response to:
identifying the first set of probability values represents the first plant, and
identifying that the second cluster of points represents the second plant positioned behind the first plant; and
actuate a treatment mechanism to treat the second plant by dispensing a treatment fluid at the second position in the row.
12. The non-transitory computer readable storage medium of claim 11 , further comprising instructions that cause the processor to:
map the first set of probability values in the probability heatmap to the first cluster of points in the three-dimensional space;
determine that the first cluster of points in the three-dimensional space does not represent the first plant based on the position of the first cluster of points in the three-dimensional space; and
responsive to determining that the first cluster of points in the three-dimensional space does not represent the first plant, update the first set of probability values in the probability heatmap to indicate that the first plant is not is present at the first position in the row.
13. The non-transitory computer readable storage medium of claim 11 , further comprising instructions that cause the processor to:
identify a subset of points of the second cluster of points in the three-dimensional space representing a particular plant feature for the second plant.
14. The non-transitory computer readable storage medium of claim 13 , further comprising instructions that cause the processor to:
update the plurality of probability values of the probability heatmap with one or more additional probability values that indicate that the particular plant feature for the second plant occurs at the second position in the row.
15. The non-transitory computer readable storage medium of claim 13 , further comprising instructions that cause the processor to:
identify, by the depth map, pixels in the image mapped by to the subset of points in the second cluster of points in the three-dimensional space as the particular feature for the second plant.
16. The non-transitory computer readable storage medium of claim 13 , further comprising instructions that cause the processor to:
remove the second cluster of points from the three-dimensional space;
responsive to removing the second cluster of points from the three-dimensional space, identify a set of probability values of the plurality of probability values corresponding to the removed second cluster of points; and
updating the identified set of probability values to indicate that the second plant is not present at positions in the row corresponding to the identified set of probability values.
17. The non-transitory computer readable storage medium of claim 11 , further comprising instructions that cause the processor to:
generate a line representing the row in in the three-dimensional space of the depth map;
for each plant of the plurality of plants in the row,
identify a cluster of points in the three-dimensional space corresponding to the plant;
determine a distance between the identified cluster of points in the three-dimensional space and the generated line using a center of mass of the identified cluster of points; and
responsive to determining that the determined distance exceeds a threshold, remove the identified cluster of points from the three-dimensional space.
18. The non-transitory computer readable storage medium of claim 11 , further comprising instructions that cause the processor to:
identify a set of probability values of the plurality of probability values representing the plant corresponding to the removed cluster of points in the three-dimensional space; and
update the identified set of probability values to indicate that no plant is present at positions in the row corresponding to the identified set of probability values.
19. The non-transitory computer readable storage medium of claim 11 , wherein instructions for identifying the second cluster of points in the three-dimensional space as the second plant at the second position in the row further cause the processor to:
for each point of the second cluster of points in the three-dimensional space, compare a position of the point in the three-dimensional space to positions of points in the first cluster of points in the three-dimensional space; and
determine that the second cluster of points in the three-dimensional space identifies the second plant based on the comparison.
20. The non-transitory computer readable storage medium of claim 11 , further comprising instructions that cause the processor to:
determine, for each pixel in the image, a depth value based on the point of the plurality of points in the three-dimensional space mapped to the pixel by the depth map; and
generate a point cloud representation of the image, wherein each point of the point cloud corresponds to a pixel in the image and is assigned the depth value determined for the corresponding pixel.
21. The non-transitory computer readable storage medium of claim 11 , wherein the farming machine does not treat the identified first plant.
22. A plant treatment system comprising:
at least one camera configured to capture an image of a plurality of plants in a row;
a treatment mechanism configured to dispense a treatment fluid at plants in the; and
an image capture system comprising computer program instructions that, when executed by a computer processor, cause the processor to:
access a probability heatmap comprising a plurality of probability values, the probability heatmap corresponding to the image, and wherein each probability value of the plurality of probability values:
corresponds to a pixel of a plurality of pixels in the image,
corresponds to a position of a plurality of positions in the row represented by the corresponding pixel in the image, and
represents a likelihood of a particular plant feature occurring at the position represented by the corresponding pixel in the image;
identify a first set of probability values in the probability heatmap representing a first plant of the plurality of plants in the row, the first plant identified based on:
a first position of the plurality of positions in the row represented by the first set of probability values, and
a first likelihood represented by the first set of probability values that the particular plant feature for the first plant occurs at the first position;
access a depth map comprising a plurality of points in a three-dimensional space, the depth map corresponding to the image of the plurality of plants in the row, and wherein:
each point in the depth map is mapped to pixels in the image, and
one or more points in the depth map corresponds to one or more positions of the plurality of positions in the row;
identify, in the depth map;
a first cluster of points of the plurality of points in the three-dimensional space representing the first plant at the first position in the row, and
a second cluster of points of the plurality of points in the three-dimensional space representing a second plant at a second position of the plurality of positions in the row, the second position behind 1 the first position relative to the farming machine;
determine the first plant obstructs the second plant in response to:
identifying the first set of probability values represents the first plant, and
identifying that the second cluster of points represents the second plant positioned behind the first plant; and
actuate a treatment mechanism to treat the second plant by dispensing a treatment fluid at the second position in the row.