Detection of plant diseases with multi-stage, multi-scale deep learning

1. A computer-implemented method of recognizing plant diseases from a plant image, comprising:

obtaining, by a processor, a first digital model for classifying an image into a class of a first set of classes corresponding to a first plurality of plant diseases, a healthy condition, or a combination of a second plurality of plant diseases;

obtaining, by the processor, a second digital model for classifying an image into a class of a second set of classes corresponding to the second plurality of plant diseases;

receiving a new image from a user device;

applying the first digital model to a plurality of first regions within the new image to obtain a plurality of classifications;

applying the second digital model to one or more second regions, each corresponding to a combination of multiple first regions of the plurality of first regions, to obtain one or more classifications,

the multiple first regions being classified into the class corresponding to the combination of the second plurality of plant diseases;

transmitting classification data related to the plurality of classifications into a class corresponding to one of the first plurality of plant diseases or the healthy condition and the one or more classifications to the user device.

2. The computer-implemented method of claim 1 , where a plant disease of the first plurality of plant diseases produces smaller lesions than a plant disease of a second plurality of plant diseases.

3. The computer-implemented method of claim 2 , the applying the first digital model comprising:

resizing the new image by a first scaling factor to obtain a resized image;

applying a first sliding window to the resized image to obtain a plurality of portions of the resized image, a size of the first sliding window being equal to a size of a sample image used to build the first digital model;

applying the first digital model to each portion of the plurality of portions of the resized image.

4. The computer-implemented method of claim 3 , further comprising:

mapping each portion of the plurality of portions of the resized image back into a region of the new image;

preparing a prediction map for the new image showing an indicator of a result of applying the first digital model to each portion of the plurality of portions of the resize image at a corresponding region of the new image;

causing a display of the prediction map.

5. The computer-implemented method of claim 3 , the applying the second digital model comprising:

combining multiple portions of the plurality of portions of the resized image, at least one portion of the multiple portions not having been classified into the first set of classes, to obtain a combined portion;

resizing the combined portion to a size of a sample image used to build the second digital model to obtain a resized combined portion;

applying the second digital model to the resized combined portion.

6. The computer-implemented method of claim 3 , the applying the second digital model comprising:

masking off regions of the plurality of first regions in the new image that were classified into the first set of classes to obtain a modified new image;

resizing the modified new image by a second scaling factor smaller than the first scaling factor to obtain a resized modified image;

applying a second sliding window to the resized modified image to obtain a plurality of portions of the resized modified image, a size of the second sliding window being equal to a size of a sample image used to build the second digital model;

applying the second digital model to each portion of the plurality of portions of the resized modified image.

7. The computer-implemented method of claim 6 , further comprising:

mapping each portion of the plurality of portions of the resized modified image back into a region of the new image;

updating a prediction map for the new image showing an indicator of a result of applying the second digital model to each portion of the plurality of portions of the resize modified image a corresponding region of the new image;

causing a display of the prediction map.

8. The computer-implemented method of claim 3 , the obtaining the first digital model comprising:

obtaining a first training set from at least a first photo showing a first symptom including a first lesion of one of a first plurality of plant diseases, a second photo showing no symptom, and a third photo showing a second symptom including a partial second lesion of one of a second plurality of plant diseases,

the first lesion being smaller than the second lesion,

the first, second, and third photos corresponding to similarly-sized fields of view;

building the first digital model from the first training set.

9. The computer-implemented method of claim 8 , the obtaining the second digital model comprising:

obtaining a second training set from at least a fourth photo showing the second symptom including third lesion, the first lesion being smaller than the third lesion;

building the second digital model from the second training set.

10. The computer-implemented method of claim 8 , the obtaining the first digital model further comprising:

determining a first image size based on the size of the first sliding window and the first scaling factor;

resizing the first photo, the second photo, or the third photo according to the first image size to obtain a first resized photo, a second resize photo, or a third resized photo.

11. The computer-implemented method of claim 1 ,

the first plurality of plant diseases including Common Rust, Eyespot, Southern Rust, or Gray Leaf Spot at an early stage,

the second plurality of plant diseases including Goss's Wilt, Northern Leaf Blight, or Gray Leaf Spot at a late stage.

12. One or more non-transitory computer-readable storage media storing instructions which when executed cause one or more processors to perform a method of recognizing plant diseases having multiple symptoms from a plant image, the method comprising:

obtaining, by a processor, a first digital model for classifying an image into a class of a first set of classes corresponding to a first plurality of plant diseases, a healthy condition, or a combination of a second plurality of plant diseases;

obtaining, by the processor, a second digital model for classifying an image into a class of a second set of classes corresponding to the second plurality of plant diseases;

receiving a new image from a user device;

applying the first digital model to a plurality of first regions within the new image to obtain a plurality of classifications;

applying the second digital model to one or more second regions, each corresponding to a combination of multiple first regions of the plurality of first regions, to obtain one or more classifications,

the multiple first regions being classified into the class corresponding to the combination of the second plurality of plant diseases;

transmitting classification data related to the plurality of classifications into a class corresponding to one of the first plurality of plant diseases or the healthy condition and the one or more classifications to the user device.

13. The one or more non-transitory computer-readable storage media of claim 12 , wherein a plant disease of the first plurality of plant diseases produces smaller lesions than a plant disease of a second plurality of plant diseases.

14. The one or more non-transitory computer-readable storage media of claim 13 , the applying the first digital model comprising:

resizing the new image by a first scaling factor to obtain a resized image;

applying a first sliding window to the resized image to obtain a plurality of portions of the resized image, a size of the first sliding window being equal to a size of a sample image used to build the first digital model;

applying the first digital model to each portion of the plurality of portions of the resized image.

15. The one or more non-transitory computer-readable storage media of claim 14 , the method further comprising:

mapping each portion of the plurality of portions of the resized image back into a region of the new image;

preparing a prediction map for the new image showing an indicator of a result of applying the first digital model to each portion of the plurality of portions of the resize image at a corresponding region of the new image;

causing a display of the prediction map.

16. The one or more non-transitory computer-readable storage media of claim 14 , the applying the second digital model comprising:

combining multiple portions of the plurality of portions of the resized image, at least one portion of the multiple portions not having been classified into the first set of classes, to obtain a combined portion;

resizing the combined portion to a size of a sample image used to build the second digital model to obtain a resized combined portion;

applying the second digital model to the resized combined portion.

17. The one or more non-transitory computer-readable storage media of claim 14 , the applying the second digital model comprising:

masking off regions of the plurality of first regions in the new image that were classified into the first set of classes to obtain a modified new image;

resizing the modified new image by a second scaling factor smaller than the first scaling factor to obtain a resized modified image;

applying a second sliding window to the resized modified image to obtain a plurality of portions of the resized modified image, a size of the second sliding window being equal to a size of a sample image used to build the second digital model;

applying the second digital model to each portion of the plurality of portions of the resized modified image.

18. The one or more non-transitory computer-readable storage media of claim 17 , the method further comprising:

mapping each portion of the plurality of portions of the resized modified image back into a region of the new image;

updating a prediction map for the new image showing an indicator of a result of applying the second digital model to each portion of the plurality of portions of the resize modified image a corresponding region of the new image;

causing a display of the prediction map.

19. The one or more non-transitory computer-readable storage media of claim 14 , the obtaining the first digital model comprising:

obtaining a first training set from at least a first photo showing a first symptom including a first lesion of one of a first plurality of plant diseases, a second photo showing no symptom, and a third photo showing a second symptom including a partial second lesion of one of a second plurality of plant diseases,

the first lesion being smaller than the second lesion,

the first, second, and third photos corresponding to similarly-sized fields of view;

building the first digital model from the first training set.

20. The one or more non-transitory computer-readable storage media of claim 19 , the obtaining the second digital model comprising:

obtaining a second training set from at least a fourth photo showing the second symptom including third lesion, the first lesion being smaller than the third lesion;

building the second digital model from the second training set.

21. The one or more non-transitory computer-readable storage media of claim 19 , the obtaining the first digital model further comprising:

determining a first image size based on the size of the first sliding window and the first scaling factor;

resizing the first photo, the second photo, or the third photo according to the first image size to obtain a first resized photo, a second resize photo, or a third resized photo.

22. The one or more non-transitory computer-readable storage media of claim 12 ,

the first plurality of plant diseases including Common Rust, Eyespot, Southern Rust, or Gray Leaf Spot at an early stage,

the second plurality of plant diseases including Goss's Wilt, Northern Leaf Blight, or Gray Leaf Spot at a late stage.