Geo-based vector data and raster data joining and/or aggregation using user defined function(s)
Techniques are disclosed herein that enable the generation and processing of user defined functions for customized aggregation and/or joining of instances of agricultural data. Various implementations include agricultural image data which includes one or more high elevation images that capture an agricultural plot. Additional or alternative implementations include transmitting the user defined function to a remote computing device for processing. In some implementations, the user defined function can be processed in tandem with additional user defined functions, where processing of instructions occurring in multiple user defined functions are not repeated.
1 . A method implemented by one or more processors, the method comprising:
identifying a first user-defined function for processing a first set of agricultural data, wherein the first set of agricultural data includes a high elevation image that includes an agricultural plot, a geo-location, a temporal value, and metadata indicating one or more crops planted in the agricultural plot, and wherein the high elevation image is divided into a plurality of cells representing one or more categories of the first set of agricultural data in the agricultural plot, wherein the first user-defined function for processing the first set of agricultural data includes generating a first index (NDVI) based on processing a plurality of instances of the first set of agricultural data, the first NDVI is a spectral imaging transformation of two or more image bands;
determining at least one portion of the plurality of cells of the high elevation image as a contiguous grouping of cells representing the same category of the agricultural data based on whether the plurality of cells exceed a threshold number of cells;
generating a boundary around the contiguous grouping of cells, wherein the first set of agricultural data of the contiguous grouping of cells is represented as first vector data and the first set of agricultural data not within the contiguous grouping of cells is represented as first raster data, wherein the first vector data represents a geometric representation of the group of cells along with the category of the group of cells;
generating a first output responsive to the first user-defined function based on the first set of agricultural data, wherein the first output includes an aggregated and/or joined agricultural data;
generating a subsequent output responsive to a second user-defined function for processing a second set of agricultural data, wherein the second set of agricultural data includes second raster data and second vector data, the second vector data indicates an unplanted portion of the agricultural plot, wherein the second user-defined function for processing the second set of agricultural data includes a second NDVI based on processing the second set of agricultural data; and
transmitting a difference between the first output and the subsequent output locally to a client device, wherein difference between the first output and the subsequent output is comparing the first NDVI and the second NDVI.
2 . The method of claim 1 , wherein the one or more categories of the first set of agricultural data are identified in the metadata of the first set of agricultural data.
3 . The method of claim 1 , wherein the first user-defined function for processing the first set of agricultural data is generated by a user of a third party application programming interface.
4 . The method of claim 3 , wherein the first output responsive to the first user-defined function includes aggregation and/or joining of one or more of an instance of the first set of agricultural data in a plurality of instances of the first set of agricultural data.
5 . The method of claim 4 , further including rendering the first output responsive to the first user-defined function via a browser of one or more display devices of the client device.
6 . The method of claim 3 , wherein generating the first output responsive to the first user-defined function includes:
generating a sequence of instructions to process the first set of agricultural data identified in the first user-defined function;
adding each instruction in the sequence of instructions to an operations queue managed by a third party;
processing one or more instructions in the operations queue;
determining whether each instruction in the sequence of instructions has been processed; and
in response to determining each instruction in the sequence of instructions has been processed, generating the first output responsive to the first user-defined function.
7 . The method of claim 6 , wherein generating the sequence of instructions to process the first set of agricultural data identified in the first user-defined function includes generating a directed acyclic graph to perform the first user-defined function.
8 . The method of claim 6 , wherein processing the one or more instructions in the operations queue includes:
identifying a batch of instructions in the operations queue; and
processing the identified batch of instructions.
9 . The method of claim 6 , wherein processing the one or more instructions in the operations queue includes processing the one or more instructions in real time as added to the operations queue.
10 . The method of claim 1 , including
transmitting to the client device instructions to implement the second set of agricultural data in the agricultural plot based on the difference.
11 . A
system comprising a processor having access to memory media storing instructions executable by the processor to perform operations, comprising:
identifying a first user-defined function for processing a first set of agricultural data, wherein the first set of agricultural data includes a high elevation image that includes an agricultural plot, a geo-location, a temporal value, and metadata indicating one or more crops planted in the agricultural plot, and wherein the high elevation image is divided into a plurality of cells representing one or more categories of the first set of agricultural data in the agricultural plot, wherein the first user-defined function for processing the first set of agricultural data includes generating a first enhanced vegetation index (NDVI) based on processing a plurality of instances of the first set of agricultural data, the first NDVI is a spectral imaging transformation of two or more image bands;
determining at least one portion of the plurality of cells of the high elevation image as a contiguous grouping of cells representing the same category of the first set of agricultural data based on whether the plurality of cells exceed a threshold number of cells;
generating a boundary around the contiguous grouping of cells, wherein the first set of agricultural data of the contiguous grouping of cells is represented as first vector data and the first set of agricultural data not within the contiguous grouping of cells is represented as first raster data, wherein the first vector data represents a geometric representation of the group of cells along with the category of the group of cells;
generating a first output responsive to the first user-defined function based on the first set of agricultural data, wherein the first output includes an aggregated and/or joined agricultural data;
generating a subsequent output responsive to a second user-defined function for processing a second set of agricultural data, wherein the second set of agricultural data includes second raster data and second vector data, the second vector data indicates an unplanted portion of the agricultural plot, wherein the second user-defined function for processing the second set of agricultural data includes a second NDVI based on processing the second set of agricultural data; and
transmitting a difference between the first output and the subsequent output locally to a client device, wherein difference between the first output and the subsequent output is comparing the first NDVI and the second NDVI.
12 . The system of claim 11 , wherein the first set of agricultural data is stored locally at the client device.
13 . The system of claim 12 , wherein the second set of agricultural data is stored at a further computing device, wherein the further computing device is remote from the client device.
14 . The system of claim 11 , further including processing the one or more instructions in an operations queue, and wherein processing the one or more instructions in the operations queue includes:
identifying a batch of instructions in the operations queue; and
processing the identified batch of instructions.
15 . The system of claim 14 , wherein processing the one or more instructions in the operations queue further includes processing the one or more instructions in real time as added to the operations queue.
16 . A non-transitory computer readable storage medium configured to store instructions that, when executed by one or more processors, cause the one or more processors to:
identify a first user-defined function for processing a first set of agricultural data, wherein the first set of agricultural data includes a high elevation image that includes an agricultural plot, a geo-location, a temporal value, and metadata indicating one or more crops planted in the agricultural plot, and wherein the high elevation image is divided into a plurality of cells representing one or more categories of the first set of agricultural data in the agricultural plot, wherein the first user-defined function for processing the first set of agricultural data includes generating a first NDVI based on processing a plurality of instances of the first set of agricultural data, the first NDVI is a spectral imaging transformation of two or more image bands;
determine at least one portion of the plurality of cells of the high elevation image as a contiguous grouping of cells representing the same category of the first set of agricultural data based on whether the plurality of cells exceed a threshold number of cells;
generate a boundary around the contiguous grouping of cells, wherein the first set of agricultural data of the contiguous grouping of cells is represented as first vector data and the first set of agricultural data not within the contiguous grouping of cells is represented as first raster data, wherein the first vector data represents a geometric representation of the group of cells along with the category of the group of cells;
generate a first output responsive to the first user-defined function based on the first set of agricultural data, wherein the first output includes an aggregated and/or joined agricultural data;
generate a subsequent output responsive to a second user-defined function for processing a second set of agricultural data, wherein the second agricultural data includes second raster data and second vector data, the second vector data indicates an unplanted portion of the agricultural plot, wherein the second user-defined function for processing the second set of agricultural data includes a second NDVI based on processing the second set of agricultural data; and
transmit a difference between the first output and the subsequent output locally to a client device, wherein difference between the first output and the subsequent output is comparing the first NDVI and the NDVI.
17 . The non-transitory computer readable storage medium of claim 16 , wherein the one or more categories of the first set of agricultural data are identified in the metadata of the first set of agricultural data.
18 . The non-transitory computer readable storage medium of claim 16 , wherein the first user-defined function for processing the first set of agricultural data is generated by a user of a third party application programming interface.
19 . The non-transitory computer readable storage medium of claim 18 , wherein the first output responsive to the first user-defined function includes aggregation and/or joining of one or more instances of the first set of agricultural data in a plurality of instances of the first set of agricultural data.