Hybrid data-compute platform

1 . A method of executing a process comprising:

registering a dataset schema with a data shipment registry, registering the dataset schema including associating one or more data shipment rules with the dataset schema;

registering the process with the data shipment registry including associating one or more processing rules with the process;

receiving a job request for the process specifying at least one dataset and retrieving the processing rules associated with the process and data shipment rules registered in association with the at least one dataset;

determining an execution location of the process based on the retrieved processing rules and the data shipment rules, the determined execution location selected from an on-premise processing location and a cloud-based processing location;

transferring, according to the data shipment rules, at least one of:

an input dataset to the determined execution location; and

an output dataset from the determined execution location;

executing the process at the determined execution location including at least one of:

when transferring according to the data shipment rules comprises transferring the input dataset to the determined execution location, using the input dataset transferred to the determined execution location; and

when transferring according to the data shipment rules comprises transferring the output dataset from the determined execution location, generating the output dataset prior to transferring the output dataset from the determined execution location;

wherein data shipment rules associated with the at least one of the input dataset and the output dataset specify requirements for modifying personally identifiable information (PII).

2 . The method of claim 1 , wherein the processing location is further determined based on one or more of:

desired performance requirements of the process;

resource constraints of the on-premise location; and

costs associated with executing the process at the cloud-based location.

3 . The method of claim 1 , wherein the data shipment rules specify one or more of:

allowed storage locations of the associated dataset;

encryption requirements for transferring the dataset; and

encryption of one or more portions of the dataset.

4 . The method of claim 1 , wherein transferring the at least one of the input dataset and the output dataset of the process comprises bulk transferring or streaming the at least one of the input dataset and the output dataset.

5 . The method of claim 1 , wherein transferring the at least one of the input dataset and the output dataset further comprises modifying the PII in the at least one of the input dataset and the output dataset.

6 . The method of claim 5 , wherein modifying the PII comprises one or more of:

removing the PII from the at least one of the input dataset and the output dataset;

replacing the PII in the at least one of the input dataset and the output dataset with non-identifiable information; and

encrypting the PII in the at least one of the input dataset and the output dataset.

7 . The method of claim 1 , wherein the data shipment rules define data security policies for the associated dataset.

8 . The method of claim 1 , wherein transferring the at least one of the input dataset and the output dataset comprises sending a notification of the transfer to an operational store for tracking locations of datasets.

9 . A system for executing a process, the system comprises:

at least one processor for executing instructions; and

at least one memory storing instructions, which when executed by the at least one processor configures the system to perform a method comprising:

registering a dataset schema with a data shipment registry, registering the dataset schema including associating one or more data shipment rules with the dataset schema;

registering the process with the data shipment registry including associating one or more processing rules with the process;

receiving a job request for the process specifying at least one dataset and retrieving the processing rules associated with the process and data shipment rules registered in association with the at least one dataset;

determining an execution location of the process based on the retrieved processing rules and the data shipment rules, the determined execution location selected from an on-premise processing location and a cloud-based processing location;

transferring, according to the data shipment rules, at least one of:

an input dataset to the determined execution location; and

an output dataset from the determined execution location;

executing the process at the determined execution location including at least one of:

when transferring according to the data shipment rules comprises transferring the input dataset to the determined execution location, using the input dataset transferred to the determined execution location; and

when transferring according to the data shipment rules comprises transferring the output dataset from the determined execution location, generating the output dataset prior to transferring the output dataset from the determined execution location,

wherein the data shipment rules comprise one or more rules for scanning the dataset for personally identifiable information (PII),

wherein transferring the at least one of the input dataset and the output dataset comprises scanning the dataset for the PII, and

wherein data shipment rules associated with the at least one of the input dataset and the output dataset specify requirements for modifying PII.

10 . The system of claim 9 , wherein the processing location is further determined based on one or more of:

desired performance requirements of the process;

resource constraints of the on-premise location; and

costs associated with executing the process at the cloud-based location.

11 . The system of claim 9 , wherein the data shipment rules specify one or more of:

allowed storage locations of the associated dataset;

encryption requirements for transferring the dataset; and

encryption of one or more portions of the dataset.

12 . The system of claim 9 , wherein transferring the at least one of the input dataset and the output dataset of the process comprises bulk transferring or streaming the at least one of the input dataset and the output dataset.

13 . The system of claim 9 , wherein transferring the at least one of the input dataset and the output dataset further comprises modifying the PII in the at least one of the input dataset and the output dataset.

14 . The system of claim 13 , wherein modifying the PII comprises one or more of:

removing the PII from the at least one of the input dataset and the output dataset;

replacing the PII in the at least one of the input dataset and the output dataset with non-identifiable information; and

encrypting the PII in the at least one of the input dataset and the output dataset.

15 . The system of claim 9 , wherein the data shipment rules define data security policies for the associated dataset.

16 . The system of claim 9 , wherein transferring the at least one of the input dataset and the output dataset comprises sending a notification of the transfer to an operational store for tracking locations of datasets.

17 . A non-transitory computer readable memory storing instructions, which when executed by at least one processor of a system configures the system to perform a method according to claim 1 .