Cross-process dispatch service in hybrid cloud scenarios
Methods, systems, and computer-readable storage media for receiving, by a cross-process scheduling engine executed within a cloud platform, work scenario metadata and sensitivity metadata, generating, by the cross-process scheduling engine, a lineage graph representing relationships between one or more of data and processes to be executed for a work scenario represented by the work scenario metadata, defining, by the cross-process scheduling engine, a set of tile segments for the work scenario based on the lineage graph and the sensitivity metadata, dispatching, by the cross-process scheduling engine, a first sub-set of tile segments for execution in a private deployment including one or more private nodes, and dispatching, by the cross-process scheduling engine, a second sub-set of tile segments for execution in a public deployment including one or more public nodes.
1. A computer-implemented method for selectively dispatching work scenarios between public deployments and private deployments, the method being executed by one or more processors and comprising:
generating, by the cross-process scheduling engine executed within a cloud platform, a lineage graph representing relationships between one or more of data and processes to be executed for a work scenario represented by work scenario metadata, the lineage graph comprising an outer layer for work scenario versioning history data representative of updates to the work scenario and an inner layer comprising a dependency graph representing up-to-date runtime lineage information for execution of the work scenario;
defining, by the cross-process scheduling engine and using the dependency graph, a set of tile segments for the work scenario based on the work scenario metadata and sensitivity metadata, each tile segment being designated as one of sensitive and non-sensitive and comprising process code, query statements, and data for execution of a portion of the work scenario, wherein a first tile segment has a dependency on a second tile segment as defined in the dependency graph;
dispatching, by the cross-process scheduling engine, a first sub-set of tile segments for execution in a private deployment comprising one or more private nodes, the first sub-set of tile segments comprising the first tile segment that is executed by a private node of the one or more private nodes;
dispatching, by the cross-process scheduling engine, a second sub-set of tile segments for execution in a public deployment comprising one or more public nodes, the second sub-set of tile segments comprising the second tile segment that is executed by a public node of the one or more public nodes; and
coordinating, by a cloud data sharing service, data flow between the private node and the public node for respective execution of the first tile segment and the second tile segment based on the dependency.
2. The method of claim 1 , further comprising:
executing a private support agent within each private node of the one or more private nodes, the private support agent providing private node metadata to the cloud data sharing service; and
executing a public support agent within each public node of the one or more public nodes, the public support agent providing public node metadata to the cloud data sharing service.
3. The method of claim 1 , wherein the sensitivity metadata comprises an indication of one or more of security-sensitivity and compliance-sensitivity of each of one or more of data to be processed in the work scenario and one or more processes of the work scenario.
4. The method of claim 1 , wherein the sensitivity metadata is provided to the cross-process scheduling engine from the cloud data sharing service that generates the sensitivity metadata based on metadata received from each of a plurality of support agent nodes executed within the private deployment and the public deployment.
5. The method of claim 1 , wherein the cross-process scheduling engine comprises a tile segment worker that selectively segments work scenarios to provide sets of tile segments.
6. The method of claim 1 , wherein defining a set of tile segments for the work scenario is executed in response to determining that the work scenario implicates one or more of security-sensitivity and compliance-sensitivity of each of one or more of data to be processed in the work scenario and one or more processes of the work scenario.
7. The method of claim 1 , wherein the public deployment comprises a public cloud and the private deployment comprises a private cloud in a hybrid cloud deployment.
8. A non-transitory computer-readable storage medium coupled to one or more processors and having instructions stored thereon which, when executed by the one or more processors, cause the one or more processors to perform operations for selectively dispatching work scenarios between public deployments and private deployments, the operations comprising:
generating, by the cross-process scheduling engine executed within a cloud platform, a lineage graph representing relationships between one or more of data and processes to be executed for a work scenario represented by work scenario metadata, the lineage graph comprising an outer layer for work scenario versioning history data representative of updates to the work scenario and an inner layer comprising a dependency graph representing up-to-date runtime lineage information for execution of the work scenario;
defining, by the cross-process scheduling engine and using the dependency graph, a set of tile segments for the work scenario based on the work scenario metadata and sensitivity metadata, each tile segment being designated as one of sensitive and non-sensitive and comprising process code, query statements, and data for execution of a portion of the work scenario, wherein a first tile segment has a dependency on a second tile segment as defined in the dependency graph;
dispatching, by the cross-process scheduling engine, a first sub-set of tile segments for execution in a private deployment comprising one or more private nodes, the first sub-set of tile segments comprising the first tile segment that is executed by a private node of the one or more private nodes;
dispatching, by the cross-process scheduling engine, a second sub-set of tile segments for execution in a public deployment comprising one or more public nodes, the second sub-set of tile segments comprising the second tile segment that is executed by a public node of the one or more public nodes; and
coordinating, by a cloud data sharing service, data flow between the private node and the public node for respective execution of the first tile segment and the second tile segment based on the dependency.
9. The non-transitory computer-readable storage medium of claim 8 , wherein operations further comprise:
executing a private support agent within each private node of the one or more private nodes, the private support agent providing private node metadata to the cloud data sharing service; and
executing a public support agent within each public node of the one or more public nodes, the public support agent providing public node metadata to the cloud data sharing service.
10. The non-transitory computer-readable storage medium of claim 8 , wherein the sensitivity metadata comprises an indication of one or more of security-sensitivity and compliance-sensitivity of each of one or more of data to be processed in the work scenario and one or more processes of the work scenario.
11. The non-transitory computer-readable storage medium of claim 8 , wherein the sensitivity metadata is provided to the cross-process scheduling engine from the cloud data sharing service that generates the sensitivity metadata based on metadata received from each of a plurality of support agent nodes executed within the private deployment and the public deployment.
12. The non-transitory computer-readable storage medium of claim 8 , wherein the cross-process scheduling engine comprises a tile segment worker that selectively segments work scenarios to provide sets of tile segments.
13. The non-transitory computer-readable storage medium of claim 8 , wherein defining a set of tile segments for the work scenario is executed in response to determining that the work scenario implicates one or more of security-sensitivity and compliance-sensitivity of each of one or more of data to be processed in the work scenario and one or more processes of the work scenario.
14. The non-transitory computer-readable storage medium of claim 8 , wherein the public deployment comprises a public cloud and the private deployment comprises a private cloud in a hybrid cloud deployment.
15. A system, comprising:
a computing device; and
a non-transitory computer-readable storage device coupled to the computing device and having instructions stored thereon which, when executed by the computing device, cause the computing device to perform operations for natural language explanations for selectively dispatching work scenarios between public deployments and private deployments, the operations comprising:
generating, by the cross-process scheduling engine executed within a cloud platform, a lineage graph representing relationships between one or more of data and processes to be executed for a work scenario represented by work scenario metadata, the lineage graph comprising an outer layer for work scenario versioning history data representative of updates to the work scenario and an inner layer comprising a dependency graph representing up-to-date runtime lineage information for execution of the work scenario;
defining, by the cross-process scheduling engine and using the dependency graph, a set of tile segments for the work scenario based on the work scenario metadata and sensitivity metadata, each tile segment being designated as one of sensitive and non-sensitive and comprising process code, query statements, and data for execution of a portion of the work scenario, wherein a first tile segment has a dependency on a second tile segment as defined in the dependency graph;
dispatching, by the cross-process scheduling engine, a first sub-set of tile segments for execution in a private deployment comprising one or more private nodes, the first sub-set of tile segments comprising the first tile segment that is executed by a private node of the one or more private nodes;
dispatching, by the cross-process scheduling engine, a second sub-set of tile segments for execution in a public deployment comprising one or more public nodes, the second sub-set of tile segments comprising the second tile segment that is executed by a public node of the one or more public nodes; and
coordinating, by a cloud data sharing service, data flow between the private node and the public node for respective execution of the first tile segment and the second tile segment based on the dependency.
16. The system of claim 15 , wherein operations further comprise:
executing a private support agent within each private node of the one or more private nodes, the private support agent providing private node metadata to the cloud data sharing service; and
executing a public support agent within each public node of the one or more public nodes, the public support agent providing public node metadata to the cloud data sharing service.
17. The system of claim 15 , wherein the sensitivity metadata comprises an indication of one or more of security-sensitivity and compliance-sensitivity of each of one or more of data to be processed in the work scenario and one or more processes of the work scenario.
18. The system of claim 15 , wherein the sensitivity metadata is provided to the cross-process scheduling engine from the cloud data sharing service that generates the sensitivity metadata based on metadata received from each of a plurality of support agent nodes executed within the private deployment and the public deployment.
19. The system of claim 15 , wherein the cross-process scheduling engine comprises a tile segment worker that selectively segments work scenarios to provide sets of tile segments.
20. The system of claim 15 , wherein defining a set of tile segments for the work scenario is executed in response to determining that the work scenario implicates one or more of security-sensitivity and compliance-sensitivity of each of one or more of data to be processed in the work scenario and one or more processes of the work scenario.