Automated plant detection using image data
A plant treatment platform uses a plant detection model to detect plants as the plant treatment platform travels through a field. The plant treatment platform receives image data from a camera that captures images of plants (e.g., crops or weeds) growing in the field. The plant treatment platform applies pre-processing functions to the image data to prepare the image data for processing by the plant detection model. For example, the plant treatment platform may reformat the image data, adjust the resolution or aspect ratio, or crop the image data. The plant treatment platform applies the plant detection model to the pre-processed image data to generate bounding boxes for the plants. The plant treatment platform then can apply treatment to the plants based on the output of the machine-learned model.
1. A method comprising:
capturing an image from a camera mounted on a plant treatment platform passing through or over a field, the captured image comprising data representing an area of the field comprising one or more individual plants in a plant row;
applying, with a computer physically coupled to the plant treatment platform, a pre-processing function to crop the captured image to generate a cropped image for processing by a plant detection model, the cropped image comprising data representing:
the one or more individual plants, and
a region surrounding the one or more individual plants, wherein the region comprises a portion of the area of the field within a threshold distance of the one or more individual plants;
inputting, with the computer, the cropped image into the plant detection model to:
identify image data of the cropped image that represents an individual plant of the one or more individual plants,
generate a plant bounding box within the cropped image that encloses the image data representing the individual plant by boxing a portion of the region surrounding the individual plant, and
identify a physical location of the individual plant based on a location of the plant bounding box in the cropped image; and
treating, with a treatment mechanism mounted on the plant treatment platform, the individual plant and the portion of the region surrounding the individual plant boxed by the plant bounding box by dispensing a treatment at the identified physical location.
2. The method of claim 1 , wherein the camera is positioned above the one or more individual plants and is directed downward.
3. The method of claim 1 , wherein the camera is positioned to a side of the one or more individual plants and is directed towards the one or more individual plants.
4. The method of claim 1 , wherein the camera is positioned above the one or more individual plants and is angled towards a direction of travel of the plant treatment platform.
5. The method of claim 1 , wherein the plant detection model comprises one or more submodels, each submodel of the one or more submodels identifying a different plant species, and wherein the plant bounding box is generated using a submodel of the one or more submodels and comprises an identifier for a plant species boxed by the plant bounding box.
6. The method of claim 1 , wherein the plant bounding box comprises a measure of confidence representing a likelihood that the plant bounding box boxes a plant in the field.
7. The method of claim 1 , wherein the plant detection model comprises a modified version of a Single Shot MultiBox Detector model.
8. The method of claim 7 , wherein the plant detection model uses at least one of following techniques: batch normalization, leaky rectified linear units, residual neural networks, custom anchor boxes, cleaned labeled data, increased spatial resolution on feature maps, spatial transformers, training loss optimization, or weighted softmax.
9. The method of claim 1 , wherein the applied pre-processing function further comprises at least one of: debayering, cropping, white balancing, resizing, exposure control, or value normalization.
10. The method of claim 9 , wherein the applied pre-processing function comprises value normalization to a fixed PPI.
11. The method of claim 1 , wherein the plant treatment platform further comprises a transportation mechanism that allows the plant treatment platform to travel through the field.
12. The method of claim 1 , further comprising:
transmitting instructions to a treatment mechanism to treat the one or more individual plants based on the generated plant bounding box, the instructions causing the treatment mechanism to:
position the treatment mechanism in a direction of the individual plant;
select a treatment fluid based on a species of the individual plant; and
dispense the treatment fluid onto the portion of the region boxed by the plant bounding box.
13. The method of claim 1 , wherein the plant detection model is trained based on labeled image data, wherein the labeled image data includes images with plant bounding boxes.
14. A computer-readable medium comprising instructions that, when executed by a processor, cause the processor to:
capture an image from a camera mounted on a plant treatment platform passing through or over a field, the captured image comprising data representing an area of the field comprising one or more individual plants in a plant row;
apply, with a computer physically coupled to the plant treatment platform, a pre-processing function to crop the captured image to generate a cropped image for processing by a plant detection model, the cropped image comprising data representing:
the one or more individual plants, and
a region surrounding the one or more individual plants, wherein the region comprises a portion of the area of the field within a threshold distance of the one or more individual plants;
input, with the computer, the cropped image into the plant detection model to:
identify image data of the cropped image that represents an individual plant of the one or more individual plants,
generate a plant bounding box within the cropped image that encloses the image data representing the individual plant by boxing a portion of the region surrounding the individual plant, and
identify a physical location of the individual plant in the field based on a location of the plant bounding box in the cropped image; and
treat, with a treatment mechanism mounted on the plant treatment platform, the individual plant and the portion of the region surrounding the individual plant boxed by the plant bounding box by dispensing a treatment at the identified physical location.
15. The computer-readable medium of claim 14 , wherein the plant detection model comprises one or more submodels, each submodel of the one or more submodels identifying a different plant species, and wherein the plant bounding box is generated using a submodel of the one or more submodels and comprises an identifier for a plant species boxed by the plant bounding box.
16. The computer-readable medium of claim 14 , wherein the plant bounding box comprises a measure of confidence representing a likelihood that the plant bounding box boxes a plant in the field.
17. The computer-readable medium of claim 14 , wherein the plant detection model uses at least one of following techniques: batch normalization, leaky rectified linear units, residual neural networks, custom anchor boxes, cleaned labeled data, increased spatial resolution on feature maps, spatial transformers, training loss optimization, or weighted softmax.
18. The computer-readable medium of claim 14 , wherein the applied pre-processing function further comprises at least one of: debayering, cropping, white balancing, resizing, exposure control, or value normalization.
19. The computer-readable medium of claim 18 , wherein the applied pre-processing function comprises value normalization to a fixed PPI.
20. The computer-readable medium of claim 14 , further comprising instructions that cause the processor to:
transmit instructions to a treatment mechanism to treat the one or more individual plants based on the generated plant bounding box, the instructions causing the treatment mechanism to:
position the treatment mechanism in a direction of the individual plant;
select a treatment fluid based on a species of the individual plant; and
dispense the treatment fluid onto the portion of the region boxed by the plant bounding box.
21. A farming machine comprising:
an imaging sensor mounted to the farming machine, the imaging sensor configured to capture an image as the farming machines passes through or over a field, the captured image comprising data representing an area of the field comprising one or more individual plants in a plant row;
a processor; and
a non-transitory, computer readable medium comprising instructions that, when executed by the processor, cause the processor to:
applying, with a computer physically coupled to the plant treatment platform, a pre-processing function to crop the captured image to generate a cropped image for processing by a plant detection model, the cropped image comprising data representing:
the one or more individual plants, and
a region surrounding the one or more individual plants, wherein the region comprises a portion of the area of the field within a threshold distance of the one or more individual plants;
inputting, with the computer, the cropped image into the plant detection model to:
identify image data of the cropped image that represents an individual plant of the one or more individual plants,
generate a plant bounding box within the cropped image that encloses the image data representing the individual plant by boxing a portion of the region surrounding the individual plant, and
identify a physical location of the individual plant in the field based on a location of the plant bounding box in the cropped image; and
treating, with a treatment mechanism mounted on the plant treatment platform, the individual plant and the portion of the region surrounding the individual plant boxed by the plant bounding box by dispensing a treatment at the identified physical location.