Granted Patent B2

US 12,450,207 · App. 17/865,945 · Granted Oct 21, 2025

Inventors: Renato Luiz De Freitas Cunha (São Paulo, BR); Roberto De Moura Estevão Filho (Teresópolis, BR); Leonardo De Oliveira Nunes (Rio de Janeiro, BR); Anirudh Badam (Issaquah, WA)

Assignee: Microsoft Technology Licensing, LLC

G06F16/212G06F16/258

Abstract

Claims (63)

1. A data processing system for processing large-scale datasets, the data processing system comprising:

at least one processor; and

a machine-readable medium storing executable instructions that, when executed, cause the processor to perform operations comprising:

receiving user input defining a workflow definition at a front-end of a workflow authoring and execution system, the workflow definition defining a plurality of operations to be performed on data from a large-scale dataset stored in a data storage system as a workflow, wherein each of the operations includes:

input metadata for each of the operations formatted according to a predefined metadata schema, the input metadata including first input metadata for a first operation of the workflow, the first input metadata identifying a location of input data in the data storage system for the first operation of the workflow;

programmed instructions for performing an atomic operation on the input data to generate output data; and

output metadata formatted according to the metadata schema, the output metadata describing the output data and identifying a location of the output data in the data storage system; and

validating a configuration of the workflow definition;

sending the workflow definition to a backend of the workflow authoring and execution system in response to the validation; and

executing the workflow with the backend of the workflow authoring and execution system by:

retrieving the input data for each of the respective operations from the location of the input data identified by the input metadata as each of the respective operations is executed;

determining a storage location for storing the output data for each of the respective operations in the data storage system as each of the respective operations is executed;

adding the determined storage location used to store the output data for each of the respective operations in the data storage system to the output metadata for each of the respective operations as the workflow is being executed;

using the output metadata from each of the respective operations as the input metadata for a next operation after each of the respective operations is executed until a last operation is reached; and

when the last operation is executed, providing the output metadata to the front-end of the workflow authoring and execution system.

2. The data processing system of claim 1 , further comprising:

retrieving the output data for the last operation from the location identified by the output metadata for the last operation; and

displaying the output data on a display device.

3. The data processing system of claim 1 , wherein each of the operations is selected from a library of predefined operations for use in authoring workflows for processing the data in the large-scale dataset.

4. The data processing system of claim 1 , further comprising:

validating the workflow definition before the workflow definition is received by the backend of the workflow authoring and execution system.

5. The data processing system of claim 1 , wherein the backend of the workflow authoring and execution system is implemented on a server of a cloud-based service, and

wherein the workflow definition is received from a client device.

6. A method of executing a workflow for processing data from a large-scale dataset stored in a data storage system, the method comprising:

receiving user input defining a workflow definition at a front-end of a workflow authoring and execution system, the workflow definition defining a plurality of operations to be performed on the data from the large-scale dataset as the workflow, wherein each of the operations includes:

input metadata for each of the operations formatted according to a predefined metadata schema, the input metadata, the input metadata including first input metadata for a first operation of the workflow, the first input metadata identifying a location of input data in the data storage system for the first operation of the workflow;

programmed instructions for performing an atomic operation on the input data to generate output data; and

output metadata formatted according to the metadata schema, the output metadata describing the output data and identifying a location of the output data in the data storage system; and

validating a configuration of the workflow definition;

sending the workflow definition to a backend of the workflow authoring and execution system in response to the validation; and

executing the workflow with the backend of the workflow authoring and execution system by:

retrieving the input data for each of the respective operations from the location of the input data identified by the input metadata as each of the respective operations is executed;

determining a storage location for storing the output data for each of the respective operations in the data storage system as each of the respective operations is executed;

adding the determined storage location used to store the output data for each of the respective operations in the data storage system to the output metadata for each of the respective operations as the workflow is being executed;

using the output metadata from each of the respective operations as the input metadata for a next operation after each of the respective operations is executed until a last operation is reached; and

when the last operation is executed, providing the output metadata to the front-end of the workflow authoring and execution system.

7. The method of claim 6 , further comprising:

retrieving the output data for the last operation from the location identified by the output metadata for the last operation; and

displaying the output data on a display device.

8. The method of claim 6 , wherein each of the operations is selected from a library of predefined operations for use in authoring workflows for processing the data in the large-scale dataset.

9. The method of claim 6 , further comprising:

validating the workflow definition before the workflow definition is received by the backend of the workflow authoring and execution system.

10. The method of claim 6 , wherein the backend of the workflow authoring and execution system is implemented on a server of a cloud-based service, and

wherein the workflow definition is received from a client device.

11. A method of authoring a workflow for processing data from a large-scale dataset stored in a data storage system, the method comprising:

defining a metadata schema for the large-scale dataset;

receiving user input at a front-end of a workflow authoring and execution system, the user input defining a workflow definition, the workflow definition including a plurality of operations to be performed as a workflow, wherein each of the operations includes:

input metadata formatted according to the metadata schema, the input metadata describing input data to be processed by the operation and identifying a location for the input data in the data storage system;

programmed instructions for performing an atomic operation on the input data to generate output data; and

output metadata formatted according to the metadata schema, the output metadata describing the output data and identifying a location for the output data in the data storage system;

receiving user input defining input parameters for the operations;

validating a configuration of the workflow definition and the input parameters; and

sending the workflow definition to a backend of the workflow authoring and execution system in response to the validation,

wherein:

the input metadata for a first operation is defined by the user input,

the backend of the workflow authoring and execution system determines a storage location for the output data of each operation as the workflow is being executed,

the determined storage location of the output data for each operation is added to the output metadata of the operation as the workflow is being executed, and

the output metadata of each operation is used as the input metadata for a subsequent operation.

12. The method of claim 11 , wherein each of the operations is selected from a library of predefined operations for use in authoring workflows for processing the data in the large-scale dataset.

13. The method of claim 11 , wherein the backend of the workflow authoring and execution system is implemented on a server of a cloud-based service, and

wherein the workflow definition is received from a client device.

14. The method of claim 11 , wherein the large-scale dataset is a geospatial dataset, and

wherein the metadata schema for the dataset is a SpatioTemporal Asset Catalogs (STAC) schema.

Assignments (1)

Continuity (1)

References Cited (14)