Blockchain integration for scalable distributed computations
An apparatus is configured to initiate distributed computations across a plurality of data processing clusters associated with respective data zones, to utilize local processing results of at least a subset of the distributed computations from respective ones of the data processing clusters to generate global processing results, and to update at least one distributed ledger maintained by one or more of the plurality of data processing clusters to incorporate one or more blocks each characterizing at least a portion of the distributed computations. Each of at least a subset of the data processing clusters is configured to process data from a data source of the corresponding data zone using one or more local computations of that data processing cluster to generate at least a portion of the local processing results. At least one of the data processing clusters is configured to apply one or more global computations to one or more of the local processing results to generate at least a portion of the global processing results.
1. A method comprising:
initiating distributed computations across a plurality of data processing clusters associated with respective data zones;
utilizing local processing results of at least a subset of the distributed computations from respective ones of the data processing clusters to generate global processing results; and
updating at least one distributed ledger maintained by one or more of the plurality of data processing clusters to incorporate one or more blocks each characterizing at least a portion of the distributed computations;
each of at least a subset of the data processing clusters being configured to process data from a data source of the corresponding data zone using one or more local computations of that data processing cluster to generate at least a portion of the local processing results;
at least one of the data processing clusters being configured to apply one or more global computations to one or more of the local processing results to generate at least a portion of the global processing results;
wherein the method is performed by at least one processing device comprising a processor coupled to a memory.
2. The method of claim 1 wherein each of at least a subset of the data processing clusters comprises a plurality of processing nodes interconnected by a network with at least one of the plurality of processing nodes comprising a ledger maintenance node.
3. The method of claim 1 wherein initiating the distributed computations comprises initiating the distributed computations from a given one of the data processing clusters and further wherein utilizing the local processing results comprises combining the local processing results in the given one of the data processing clusters.
4. The method of claim 1 wherein the distributed ledger is created responsive to the initiating of the distributed computations.
5. The method of claim 1 wherein the distributed ledger is collectively maintained by the plurality of data processing clusters and comprises a plurality of blocks that are sequentially added to the distributed ledger by respective ones of the data processing clusters of respective ones of the data zones.
6. The method of claim 1 wherein a given block added to the distributed ledger comprises information characterizing one or more local computations performed by a given one of the data processing clusters.
7. The method of claim 1 wherein a given block added to the distributed ledger comprises information characterizing one or more global computations performed by a given one of the data processing clusters.
8. The method of claim 1 wherein a given block added to the distributed ledger comprises one or more of:
information identifying a corresponding one of the data processing clusters; and
information identifying a corresponding one of the data zones.
9. The method of claim 1 wherein a given block added to the distributed ledger comprises one or more of:
information identifying particular local code utilized by a given one of the data processing clusters in performing one or more local computations; and
information identifying particular global code utilized by a given one of the data processing clusters in performing one or more global computations.
10. The method of claim 1 wherein a given block added to the distributed ledger comprises one or more of:
at least one of the local processing results; and
at least one of the global processing results.
11. The method of claim 1 wherein a given block added to the distributed ledger comprises one or more of:
an identifier of an entity initiating the distributed computations;
a timestamp associated with initiation of the distributed computations;
identifiers of respective data zones participating in the distributed computations;
identifiers of respective data sources providing data for one or more local computations; and
identifiers of respective meta-resources to be utilized for one or more local computations.
12. The method of claim 1 wherein a given block added to the distributed ledger is also added to one or more other distributed ledgers.
13. The method of claim 1 wherein a plurality of distributed ledgers are maintained for respective ones of the data zones.
14. The method of claim 1 wherein a plurality of distributed ledgers are maintained for respective datasets utilized in respective ones of the data zones in performing the local computations.
15. A computer program product comprising a non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code when executed by at least one processing device causes said at least one processing device:
to initiate distributed computations across a plurality of data processing clusters associated with respective data zones;
to utilize local processing results of at least a subset of the distributed computations from respective ones of the data processing clusters to generate global processing results; and
to update at least one distributed ledger maintained by one or more of the plurality of data processing clusters to incorporate one or more blocks each characterizing at least a portion of the distributed computations;
each of at least a subset of the data processing clusters being configured to process data from a data source of the corresponding data zone using one or more local computations of that data processing cluster to generate at least a portion of the local processing results;
at least one of the data processing clusters being configured to apply one or more global computations to one or more of the local processing results to generate at least a portion of the global processing results.
16. The computer program product of claim 15 wherein the distributed ledger is created responsive to the initiating of the distributed computations.
17. The computer program product of claim 15 wherein the distributed ledger is collectively maintained by the plurality of data processing clusters and comprises a plurality of blocks that are sequentially added to the distributed ledger by respective ones of the data processing clusters of respective ones of the data zones.
18. An apparatus comprising:
at least one processing device having a processor coupled to a memory;
wherein said at least one processing device is configured:
to initiate distributed computations across a plurality of data processing clusters associated with respective data zones;
to utilize local processing results of at least a subset of the distributed computations from respective ones of the data processing clusters to generate global processing results; and
to update at least one distributed ledger maintained by one or more of the plurality of data processing clusters to incorporate one or more blocks each characterizing at least a portion of the distributed computations;
each of at least a subset of the data processing clusters being configured to process data from a data source of the corresponding data zone using one or more local computations of that data processing cluster to generate at least a portion of the local processing results;
at least one of the data processing clusters being configured to apply one or more global computations to one or more of the local processing results to generate at least a portion of the global processing results.
19. The apparatus of claim 18 wherein the distributed ledger is created responsive to the initiating of the distributed computations.
20. The apparatus of claim 18 wherein the distributed ledger is collectively maintained by the plurality of data processing clusters and comprises a plurality of blocks that are sequentially added to the distributed ledger by respective ones of the data processing clusters of respective ones of the data zones.