EFFICIENT GENERATION OF ZONAL SUMMARIES OF GEOSPATIAL DATA
The method includes a first node receiving an input shape representing a field boundary with geographic coordinates. The first node accesses a geospatial data raster which includes tiles and with metadata defining boundaries of the tiles. The input shape is partitioned into multiple request shapes based on the boundaries of the tiles and the input shape's coordinates. The first node generates a first request to a second node to produce a zonal summary for the input shape. The second node generates a second request in response to receiving the first request, to generate a request shape digest for each of the plurality of request shapes. The second node instantiates workers to perform the second request. The workers generate digests based on tiles overlapping the request shape, and provides the digests to the first computing node, which aggregates the digests to produce a zonal summary for the input shape.
1 . A method comprising:
receiving, by a first computing node, an input shape having geographic coordinates corresponding to a field boundary in a geographic area;
accessing, by the first computing node, a geospatial data raster and metadata corresponding to the geospatial data raster, the geospatial data raster comprising a plurality of tiles, and the metadata defining geospatial boundaries of the plurality of tiles;
partitioning, by the first computing node, the input shape into a plurality of request shapes representing the input shape according to the geospatial boundaries of the plurality of tiles and the geographic coordinates of the input shape;
generating, at the first computing node, one or more first requests to generate a zonal summary for the input shape, a first request specifying at least the geospatial data raster and the plurality of request shapes;
generating, at each of one or more second computing nodes receiving the first request, a second request to generate a request shape digest for each of the plurality of request shapes, the second request configured to be executed by one or more workers instantiated on the one or more second computing nodes;
generating, by the one or more workers instantiated on the one or more second computing nodes and responsive to receiving the second request, a request shape digest for each of the plurality of request shapes based on tiles of the geospatial data raster overlapping the request shape; and
generating, at the first computing node and responsive to completion of the second request by the one or more workers, a zonal summary for the input shape by aggregating zone digests for the plurality of request shapes representing the input shape.
2 . The method of claim 1 , wherein the input shape is partitioned into the plurality of request shapes representing the input shape based on a maximum execution time.
3 . The method of claim 1 , wherein each instantiated worker generates a request shape digest for a same number of request shapes.
4 . The method of claim 1 , wherein a number of workers instantiated on the one or more second computing nodes corresponds to a number of request shapes.
5 . The method of claim 1 , wherein a number of workers instantiated on the one or more second computing nodes is determined by a cost-based analysis based on an availability of resources of the one or more second computing nodes.
6 . The method of claim 1 , wherein generating the first request comprises grouping the request shapes into one or more requests based on proximity of the geographic coordinates of the request shapes.
7 . The method of claim 6 , wherein grouping the request shapes comprises constructing a graph among the request shapes and identifying strongly connected components by applying Kosaraju's algorithm.
8 . The method of claim 1 , wherein the first request further comprises a date and wherein the worker retrieves the plurality of tiles based on the date.
9 . The method of claim 1 , further comprising:
selecting each of the one or more second computing nodes, by the first computing node, based on locality between each of the one or more second computing nodes and the plurality of tiles of the one or more geospatial data raster.
10 . A computer system, comprising:
a first database comprising one or more geospatial data raster, the one or more geospatial data raster comprising a plurality of tiles;
a second database comprising field data, the field data comprising a plurality of input shapes;
a first computing node comprising:
one or more computer processors, and
one or more memories comprising stored instructions that when executed by the one or more computer processors causes the first computing node to:
receive an input shape having geographic coordinates corresponding to a field boundary in a geographic area;
access a geospatial data raster and metadata corresponding to the geospatial data raster, the geospatial data raster comprising a plurality of tiles, and the metadata defining geospatial boundaries of the plurality of tiles;
partition the input shape into a plurality of request shapes representing the input shape according to the geospatial boundaries of the plurality of tiles and the geographic coordinates of the input shape;
generate a first request to generate a zonal summary for the input shape, the first request specifying at least the geospatial data raster and the plurality of request shapes; and
generate, responsive to completion of a second request by one or more workers instantiated on one or more second computing nodes, a zonal summary for the input shape by aggregating zone digests for the plurality of request shapes representing the input shape; and
the one or more second computing nodes comprising:
one or more computer processors; and
one or more memories including stored instructions, that when executed by the one or more computer processors, cause each of the one or more second computing nodes to:
generate, responsive to receiving the first request, a second request to generate a request shape digest for each of the plurality of request shapes, the second request configured to be executed by one or more workers instantiated on the one or more second computing nodes; and
generate, by the one or more workers instantiated on the one or more second computing nodes and responsive to receiving the second request, a request shape digest for each of the plurality of request shapes based on tiles of the geospatial data raster overlapping the request shape.
11 . The computer system of claim 10 , wherein the input shape is partitioned into the plurality of request shapes representing the input shape based on a maximum execution time.
12 . The computer system of claim 10 , wherein each instantiated worker generates a request shape digest for a same number of request shapes.
13 . The computer system of claim 10 , wherein a number of workers instantiated on the one or more second computing nodes corresponds to a number of request shapes.
14 . The computer system of claim 10 , wherein a number of workers instantiated on the one or more second computing nodes is determined by a cost-based analysis based on an availability of resources.
15 . The computer system of claim 10 , wherein generating the first request comprises grouping the request shapes into one or more requests based on proximity of the geographic coordinates of the request shapes.
16 . The computer system of claim 15 , wherein grouping the request shapes comprises constructing a graph among the request shapes and identifying strongly connected components by applying Kosaraju's algorithm.
17 . The computer system of claim 10 , wherein the first request further comprises a date and wherein the worker retrieves the plurality of tiles based on the date.
18 . The computer system of claim 10 , further comprising:
selecting each of the one or more second computing nodes, by the first computing node, based on locality between each of the one or more second computing nodes and the plurality of tiles of the one or more geospatial data raster.
19 . A non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to:
receive an input shape having geographic coordinates corresponding to a field boundary in a geographic area;
access a geospatial data raster and metadata corresponding to the geospatial data raster, the geospatial data raster comprising a plurality of tiles, and the metadata defining geospatial boundaries of the plurality of tiles;
partition the input shape into a plurality of request shapes representing the input shape according to the geospatial boundaries of the plurality of tiles and the geographic coordinates of the input shape;
generate a first request to generate a zonal summary for the input shape, the first request specifying at least the geospatial data raster and the plurality of request shapes; and
generate, responsive to completion of a second request by one or more workers, a zonal summary for the input shape by aggregating zone digests for the plurality of request shapes representing the input shape.
20 . The non-transitory computer-readable medium of claim 19 , wherein the input shape is partitioned into the plurality of request shapes representing the input shape based on a maximum execution time.