Determination of droplet characteristics
A system and method for the detection and tracking of droplets sprayed from an agricultural spray nozzle is provided. The system includes a sensor configured to observe droplets sprayed from the nozzle, a frame extractor module, a droplet shape and size extraction module, a droplet tracking module, and a data log module. The system may provide an artificial intelligence (AI)-enabled framework capable of processing images obtained of droplets, detecting and tracking all droplets appearing across image frames, and determining the droplets' geometric and dynamic data. The system further provides for the integration of deep-learning techniques into an image processing algorithm which enables precise and reliable determination of droplet characteristics. In addition, the deep-learning framework produces consistent results under a variety of uncertain imaging conditions.
1 . A system for detection and tracking of droplets sprayed from a nozzle, the system comprising:
a sensor configured to observe droplets sprayed from the nozzle;
a frame extractor module configured to obtain data from the sensor;
a droplet shape and size extraction module configured to extract information from the data obtained by the frame extractor module;
a droplet tracking module configured to track individual droplets using the information extracted by the droplet shape and size extraction module; and
a data log module configured to store a list of droplets and information associated with the droplets,
wherein the droplet tracking module is configured to update values stored in the data log module, including determining whether a current position of an individual droplet is outside an image frame, and when the current position is outside the image frame, removing the individual droplet and information associated with the individual droplet from the data log module.
2 . The system of claim 1 , wherein the frame extractor module is configured to extract frames from an input video at a fixed frame rate as defined by a user.
3 . The system of claim 1 , wherein the droplet shape and size extraction module is configured to extract geometric information about detected droplets in a current frame provided by the frame extractor module.
4 . The system of claim 1 , wherein the droplet tracking module is configured to track droplets by matching the droplets between frames based upon information obtained from a new frame which has been processed by the droplet shape and size extraction module.
5 . The system of claim 1 , wherein the data log module is configured to store a list of droplets and information associated with the droplets, including one or more of: position in a current frame, position in a previous frame, shape and size of each droplet in the current frame, parameters of a dynamic model associated with each droplet, a numerical value that counts a number of times each droplet has been matched continuously with previous frames referred to as success value, a numerical value that counts a number of times each droplet has not been matched with previous frames referred to as loss value, or a unique identifier for each droplet.
6 . The system of claim 3 , wherein the droplet shape and size extraction module is configured to output droplet shape information, the droplet shape and size extraction module including:
a frame noise suppression module using a Gaussian filter;
an object detection module configured to determine a detected droplets confidence score and corresponding bounding box coordinates by executing a trained deep neural network model;
a refinement module configured to select droplets having a confidence value above a user-defined threshold;
a least square-based ellipse fitting module configured to provide information associated with a fitted ellipse for the selected droplets; and
an extraction module configured to extract information by fitting an ellipse to detected droplets and transmit the information to the droplet tracking module.
7 . The system of claim 3 , wherein the droplet shape and size extraction module includes a neural network configured to refine droplet image data, the droplet shape and size extraction module including:
a feature extraction module including a deep neural network with convolution operations that learns to recognize important features in an image to allow the deep neural network to distinguish droplets;
a region proposal network including a second neural network that receives an output from the feature extraction module and identifies and outputs proposed regions in a feature map that contain an object;
a region of interest (ROI) pooling module configured to make all of the proposed regions a same size; and
an inference network including connected layers that take reshaped region proposal regions as an input and output the droplets and their corresponding bounding box, and further outputs a confidence score for each of the detected droplets in the image, wherein the confidence score is a numerical value within a range of 0 and 1 that is configured to filter out detections having confidence scores below a user-defined threshold, which are decarded as not being droplets.
8 . The system of claim 6 , wherein accuracy of the trained model is evaluated using images in a set Ω as defined in an equation:
1
n
(
Ω
)
∑
x
=
1
n
(
Ω
)
T
Dx
T
Dx
+
F
Dx
+
M
Dx
where a total number of images in the set 22 is represented using n(Ω), M DX is a total number of droplets missed by the trained model for an image when compared to its ground truth, Fox is a total number of incorrect detections made by the trained model for the image when compared to its ground truth, T DX is a true detection which is computed by subtracting a total number of droplets detected by the trained model for the image with the total number of incorrect detections for the image.
9 . The system of claim 8 , wherein the ground truth includes manually labeled data used for network training, including a bounding box around each droplet in the image.
10 . The system of claim 8 , wherein the accuracy of the trained model is configured to be used to guide a training process of the trained model, and wherein the training process of the model is stopped when the accuracy is greater than or equal to 90%.
11 . The system of claim 10 , wherein the training process is not limited to real data of droplets, and the training process is configured to use generated images of droplets using generative models during the training of the neural network model used for droplet detection.
12 . The system of claim 1 , wherein the tracking module includes a multi-step process comprising droplet matching and sharing of outputs with the data log module.
13 . The system of claim 12 , wherein the droplet matching includes:
computing a cost between new droplets and past droplets that are stored in the data log module;
applying a Hungarian algorithm to determine a unique match between new droplets and past droplets that are stored in data log module; and
providing refinement and grouping of droplets that are matched between past droplets and new droplets obtained by applying the Hungarian algorithm.
14 . The system of claim 13 , wherein the refinement and grouping of droplets that are matched between past droplets and new droplets includes:
for each match between new droplets and past droplets:
if the cost is below a user-defined threshold, accept each pair and update information for matched droplets based upon the information from the new droplet; and
if the cost is above the user-defined threshold then:
for unmatched droplets stored in the data log module: update information for unmatched droplets using a Kalman filter for predicting a position of the unmatched droplets in a next frame; and
for unmatched droplets detected in a current frame:
append each of the unmatched droplets detected in the current frame into the data log if a number of detected droplets in the current frame is greater than the number of droplets in the data log.
15 . The system of claim 14 , wherein updating information for matched droplets includes:
copy current location information of a past droplet into past location information of the past droplet, including replacing the current location information of a past droplet with location information of a new droplet;
update shape and size of each droplet of the past droplet with the shape and size information of the new droplet;
update parameters of a dynamic model stored for the past droplet;
increase a first numerical value by one that counts a number of times the past droplet has been matched continuously in previous frames; and
update a second numerical value to zero that counts a number of times the past droplet has not been matched continuously in previous frames.
16 . The system of claim 14 , wherein updating information for unmatched droplets includes the following steps for each droplet pair of a new droplets and a past droplet:
copy current location information of the past droplet into past location information of the past droplet, and replace the current location information with a Kalman filter prediction location of each droplet in the next frame;
update shape and size of each droplet of the past droplet with the shape and size information of the new droplet;
update a numerical value to zero that counts a number of times the past droplet has been matched continuously in previous frames; and
update a loss value with an increment of one.
17 . The system of claim 14 , wherein the appending includes appending information into the data log module associated with each droplet, including:
position in the current frame is stored based upon the information extracted from droplet shape and size extraction module;
position in a previous frame is left empty;
shape and size of each droplet in the current frame is stored based upon the information extracted from droplet shape and size extraction module;
parameters of a dynamic model associated with each droplet use the dynamic model;
a first numerical value that counts a number of times each droplet has been matched continuously with previous frames referred to as success value is set to zero;
a second numerical value that counts a number of times each droplet has not been matched with previous frames referred to as loss value is set to zero; and
a unique identifier for each droplet is added.
18 . The system of claim 1 , wherein the system is configured to repetitively detect and track droplets for image frames extracted by a frame extractor.
19 . The system of claim 1 , further comprising a nozzle controller configured to control function of the nozzle to implement feedback control of nozzle output based on the information associated with the droplets.
20 . The system of claim 1 , wherein at least a portion of the system is incorporated into an agricultural machine.
21 . A method for detection and tracking of droplets sprayed from a nozzle, the method comprising:
observing droplets sprayed from the nozzle using one or more sensors;
obtaining data from the one or more sensors by examining a frame from the one or more sensors;
extracting information from the data obtained from the one or more sensors regarding a droplet shape and size;
tracking individual droplets using the information extracted regarding the droplet shape and size;
storing a list of droplets and information associated with the droplets in a data log; and
updating values stored in the data log, including determining whether a current position of an individual droplet is outside an image frame, and when the current position is outside the image frame, removing the individual droplet and information associated with the individual droplet from the data log.
22 . The method of claim 21 , further comprising repetitively detecting and tracking droplets from image frames.
23 . The method of claim 21 , further comprising tracking droplets by matching the droplets between frames based upon information obtained from a new frame.