Observing crop growth through embeddings
Implementations are described herein for reducing the time and costs associated with the collection and processing of information for observing and evaluating crop growth. In various implementations, a temporal sequence of images depicting a growth of a crop over a time interval may be processed using a machine learning model. Based on the processing, a crop trajectory of image embeddings may be generated that represent the growth of the crop over the time interval. The crop trajectory of image embeddings may be compared with one or more reference crop trajectories of image embeddings. Each of the one or more reference crop trajectories may include a plurality of image embeddings that represent growth of the same type of crop as the crop trajectory of image embeddings over a respective time interval. Data associated with the comparing may be provided as output.
1 . A method implemented using one or more processors, the method comprising:
capturing, using an image capturing device of an agricultural robot travelling through a field, a temporal sequence of images depicting a growth of a crop in the field;
processing, on the agricultural robot as it travels through the field, a temporal sequence of images depicting the growth of the crop in the field over a time interval using a machine learning model, the machine learning model:
generating a crop trajectory of image embeddings that represent the growth of the crop over the time interval, the crop trajectory comprising at least a first stage trajectory representing a first growth stage of the crop in the field and a second stage trajectory representing a second growth stage of the crop in the field;
comparing stages of the crop trajectory of image embeddings with stages of one or more reference crop trajectories of image embeddings to identify a trajectory deviation and a stage deviation between the crop trajectory and each of the one or more reference crop trajectories, wherein:
each of the one or more reference crop trajectories includes a plurality of image embeddings that represent growth of a same type of crop as the crop trajectory of image embeddings over a respective time interval, and
each reference crop trajectory comprises one or more stages representing the first growth stage and the second growth stage of the crop;
identifying, based on the comparison and the identified stage deviation, a cause from a set of causes that induce the trajectory deviation between the crop trajectory and the one or more reference crop trajectories; and
providing, to a client device remote from the agricultural robot, data associated with the identified cause for display on the client device.
2 . The method of claim 1 , wherein the causing includes causing a display to simultaneously render the crop trajectory of image embeddings and one or more of the reference crop trajectories.
3 . The method of claim 2 , wherein causing the display to simultaneously render the trajectories includes implementing a t-distributed stochastic neighbor embedding (t-SNE) technique to render the trajectories in two or three dimensions.
4 . The method of claim 1 , further comprising generating one or more phenotypic conclusions about the growth of the crop based on the comparing, wherein the data associated with the comparing includes the one or more phenotypic conclusions.
5 . The method of claim 4 , wherein the generating includes applying a classifier machine learning model to the crop trajectory of image embeddings, wherein the classifier machine learning model is trained based at least in part based on the one or more reference trajectories.
6 . The method of claim 5 , wherein the classifier machine learning model comprises a recurrent neural network.
7 . The method of claim 1 , wherein the comparing includes performing regression analysis on the crop trajectory of image embeddings and one or more of the reference crop trajectories.
8 . The method of claim 1 , further comprising:
determining, based on the comparison of the identified stage deviation and the corresponding identified cause, an action to implement in the field to remedy the cause; and
scheduling the action to implement the field by the agricultural robot.
9 . The method of claim 1 , further comprising:
implementing, in the field by the agricultural robot, the action to remedy the identified cause.
10 . The method of claim 1 , wherein processing the temporal sequence of images depicting the growth of the crop in the field over the time interval using the machine learning model further comprises:
providing, for display on the client device, a first visual representing the crop trajectory;
providing, for display on the client device, a second visual representing the crop trajectory compared to reference crop trajectories;
receiving, from the client device, a user input corresponding to the stage deviation; and
wherein identifying the cause is additionally based on the user input.
11 . The method of claim 1 , wherein providing, for display on the client device, the first visual representing the crop trajectory and the second visual representing the crop trajectory compared to reference crop trajectory comprises applying on or more functions to the image embeddings to reduce their dimensionality for visualization on the client device.
12 . A method implemented using one or more processors, the method comprising:
accessing, from an image capturing device of an agricultural robot travelling through a field, a temporal sequence of images depicting a growth of a crop in the field;
processing, on the agricultural robot as it travels through the field, a temporal sequence of images depicting the growth of the crop in the field over a time interval using a machine learning model, the machine learning model:
generating a crop trajectory of image embeddings that represent the growth of the crop over the time interval, the crop trajectory comprising at least a first stage trajectory representing a first growth stage of the crop in the field and a second stage trajectory representing a second growth stage of the crop in the field;
comparing stages of the crop trajectory of image embeddings with stages of one or more reference crop trajectories of image embeddings to identify a trajectory deviation and a stage deviation between the crop trajectory and each of the one or more reference crop trajectories, wherein:
each of the one or more reference crop trajectories includes a plurality of image embeddings that represent growth of a same type of crop as the crop trajectory of image embeddings over a respective time interval, and
each of the one or more reference crop trajectories includes a plurality of image embeddings that represent growth of a same type of crop as the crop trajectory of image embeddings over a respective time interval, and
identifying, based on the comparison and the identified stage deviation, a cause from a set of causes that induce the trajectory deviation between the crop trajectory and the one or more reference crop trajectories; and
causing a display to simultaneously render the crop trajectory of image embeddings, one or more of the reference crop trajectories, and the cause of the set of causes, the displayed reference crop trajectories and the crop trajectory of image embeddings illustrating the trajectory deviation.
13 . The method of claim 12 , wherein causing the display to simultaneously render the trajectories includes implementing a t-distributed stochastic neighbor embedding (t-SNE) technique or principal component analysis (PCA) to render the trajectories in two or three dimensions.
14 . The method of claim 12 , further comprising generating one or more phenotypic conclusions about the growth of the crop based on the identifying, wherein data associated with the identifying includes the one or more phenotypic conclusions.
15 . The method of claim 14 , wherein the generating includes applying a classifier machine learning model to the crop trajectory of image embeddings, wherein the classifier machine learning model is trained based at least in part based on the one or more reference trajectories.
16 . The method of claim 15 , wherein the classifier machine learning model comprises a recurrent neural network.
17 . The method of claim 12 , wherein the identifying includes performing regression analysis on the crop trajectory of image embeddings and one or more of the reference crop trajectories.
18 . A system comprising one or more processors and memory storing instructions that, in response to execution of the instructions, cause the one or more processors to:
capture, using an image capturing device of an agricultural robot travelling through a field, a temporal sequence of images depicting a growth of a crop in the field;
process, on the agricultural robot, a temporal sequence of images depicting the growth of the crop in the field over a time interval using a machine learning model, the machine learning model:
generating a crop trajectory of image embeddings that represent the growth of the crop over the time interval, the crop trajectory comprising at least a first stage trajectory representing a first growth stage of the crop in the field and a second stage trajectory representing a second growth stage of the crop in the field;
comparing stages of the crop trajectory of image embeddings with stages of one or more reference crop trajectories of image embeddings to identify a trajectory deviation and a stage deviation between the crop trajectory and each of the one or more reference crop trajectories, wherein:
each of the one or more reference crop trajectories includes a plurality of image embeddings that represent growth of a same type of crop as the crop trajectory of image embeddings over a respective time interval, and
each reference crop trajectory comprises one or more stages representing the first growth stage and the second growth stage of the crop;
identifying, based on the comparison and the identified stage deviation, a cause from a set of causes that induce the trajectory deviation between the crop trajectory and the one or more reference crop trajectories; and
provide, to a client device remote from the agricultural robot, data associated with the identified cause for display on the client device.
19 . The system of claim 18 , wherein the instructions include instructions to implement a t-distributed stochastic neighbor embedding (t-SNE) technique to render the trajectories in two or three dimensions.
20 . The system of claim 18 , wherein the instructions include instructions to generate one or more phenotypic conclusions about the growth of the crop based on the comparing, wherein the data associated with the comparing includes the one or more phenotypic conclusions.