Fusing multimodal environmental data for agricultural inference
Implementations are disclosed for fusing multiple modalities of data into a multimodal feature embedding and then processing the multimodal feature embedding using various downstream processes for training and/or inference purposes. In various implementations, multiple different modalities of agricultural data about an agricultural parcel may be obtained. Each modality of agricultural data may be processed based on a respective modality-specific encoder to generate a respective modality-specific embedding. The plurality of modality-specific embeddings may be processed based on a multimodal fusion machine learning model to generate a multimodal feature embedding that represents the agricultural parcel. In some implementations, the multimodal feature embedding may be processed using downstream computer process(es) to generate agricultural prediction(s) about the agricultural parcel. Additionally or alternatively, the multimodal feature embedding may be used to train the multimodal fusion model and/or the modality specific encoder(s).
1 . A method implemented using one or more processors, comprising:
obtaining multiple different modalities of agricultural data about an agricultural parcel corresponding to a first time;
processing a first modality of agricultural data based on a first modality-specific encoder to generate a first modality-specific embedding and a second modality of agricultural data based on a second modality-specific encoder to generate a second modality-specific embedding, wherein the first and second modality-specific encoders are pre-trained for the first and second modalities of agricultural data, respectively, using masked autoencoding;
processing a plurality of the modality-specific embeddings based on a multimodal fusion machine learning model to generate a multimodal feature embedding that represents the agricultural parcel, the plurality of the modality-specific embeddings including the first modality-specific embedding and the second modality-specific embedding;
generating a third modality of agricultural data using the multimodal feature embedding, the third modality of agricultural data different than the multiple different modalities of agricultural data;
generating, using one or more downstream computer processes, one or more agricultural predictions about the agricultural parcel based on the multimodal feature embedding, the one or more agricultural predictions to include agricultural data corresponding to a second time, the second time different than the first time; and
causing one or more computing devices to render output that includes one or more of the agricultural predictions.
2 . The method of claim 1 , wherein the multimodal fusion machine learning model is jointly trained with at least some of the first or second modality-specific encoders.
3 . The method of claim 2 , wherein the multimodal fusion machine learning model is jointly trained using the masked autoencoding.
4 . The method of claim 1 , wherein the multimodal fusion machine learning model includes a transformer.
5 . The method of claim 1 , wherein the multiple different modalities of data include at least one modality that includes agricultural time series data about the agricultural parcel.
6 . The method of claim 5 , wherein the agricultural time series data about the agricultural parcel includes soil moisture data.
7 . The method of claim 5 , wherein the agricultural time series data about the agricultural parcel includes climate data.
8 . The method of claim 1 , wherein the multiple different modalities of data include at least one modality that includes tabular data about the agricultural parcel.
9 . The method of claim 8 , wherein the tabular data includes soil properties of the agricultural parcel.
10 . The method of claim 1 , wherein the multiple different modalities of data include at least one modality that includes satellite or aerial imagery of the agricultural parcel.
11 . The method of claim 1 , wherein one or more of the downstream computer processes includes identifying one or more reference multimodal feature embeddings that are sufficiently proximate to the multimodal feature embedding in an embedding space, wherein the one or more reference multimodal feature embeddings were generated by processing the multiple different modalities of agricultural data about one or more reference agricultural parcels.
12 . The method of claim 11 , wherein the output includes a recommendation of a suitable crop for the agricultural parcel, wherein the suitable crop is selected based on having been grown in the one or more reference agricultural parcels.
13 . The method of claim 1 , wherein one or more of the downstream computer processes includes processing the multimodal feature embedding using a downstream machine learning model to perform multi-crop yield forecasting for the agricultural parcel.
14 . A system comprising one or more processors and memory storing instructions that, in response to execution by the one or more processors, cause the one or more processors to:
obtain multiple different modalities of agricultural data about an agricultural parcel corresponding to a first time;
process a first modality of agricultural data based on a first modality-specific encoder to generate a first modality-specific embedding and a second modality of agricultural data based on a second modality-specific encoder to generate a second modality-specific embedding, wherein the first and second modality-specific encoders are pre-trained for the first and second modalities of agricultural data, respectively, using masked autoencoding;
process a plurality of the modality-specific embeddings based on a multimodal fusion machine learning model to generate a multimodal feature embedding that represents the agricultural parcel, the plurality of the modality-specific embeddings including the first modality-specific embedding and the second modality-specific embedding;
generate a third modality of agricultural data using the multimodal feature embedding, the third modality of agricultural data different than the multiple different modalities of agricultural data;
generate, using one or more downstream computer processes, one or more agricultural predictions about the agricultural parcel based on the multimodal feature embedding, the one or more agricultural predictions to include agricultural data corresponding to a second time, the second time different than the first time; and
cause one or more computing devices to render output that includes one or more of the agricultural predictions.
15 . The system of claim 14 , wherein the multimodal fusion machine learning model is jointly trained with at least some of the first or second modality-specific encoders.
16 . The system of claim 15 , wherein the multimodal fusion machine learning model is jointly trained using the masked autoencoding.
17 . The system of claim 14 , wherein the multimodal fusion machine learning model includes a transformer.
18 . The system of claim 14 , wherein the multiple different modalities of data include at least one modality that includes agricultural time series data about the agricultural parcel.
19 . A method implemented using one or more processors, comprising:
obtaining multiple different modalities of agricultural data about an agricultural parcel corresponding to a first time;
masking one or more of the multiple different modalities of agricultural data;
processing a remaining first modality of agricultural data based on a first modality-specific encoder to generate a first modality-specific embedding and a remaining second modality of agricultural data based on a second modality-specific encoder to generate a second modality-specific embedding, wherein the first and second modality-specific encoders are pre-trained for the remaining first and second modalities of agricultural data, respectively, using masked autoencoding;
processing a plurality of the modality-specific embeddings based on a multimodal fusion machine learning model to generate a multimodal feature embedding that represents the agricultural parcel, the plurality of the modality-specific embeddings including the first modality-specific embedding and the second modality-specific embedding;
generating a third modality of agricultural data using the multimodal feature embedding, the third modality of agricultural data different than the multiple different modalities of agricultural data;
generating, using one or more downstream computer processes, one or more agricultural predictions about the agricultural parcel based on the multimodal feature embedding, the one or more agricultural predictions to include agricultural data corresponding to a second time, the second time different than the first time;
comparing the one or more agricultural predictions to one or more ground truth observations; and
training the multimodal fusion machine learning model based on the comparing.
20 . The method of claim 19 , further including jointly training one or more of the first or second modality-specific encoders based on the comparing.