Decentralized file system and message bus architecture for processing training sets in multi-cloud computing environment
In a multi-cloud computing environment comprising a plurality of cloud platforms, wherein one cloud platform is a source of a model and a data set and further wherein the model is to be executed against the data set on one or more of the other cloud platforms, the method maintains a decentralized architecture comprising a file system and a message bus, wherein the file system comprises a plurality of decentralized file system nodes corresponding to the plurality of cloud platforms, and the message bus comprises a plurality of decentralized message bus nodes corresponding to the plurality of cloud platforms. Further, the method manages sharing of the model and the data set via at least a portion of the decentralized file system nodes and manages messaging related to execution of the model against the data set via at least a portion of the decentralized message bus nodes.
1. A method comprising:
in a given cloud platform comprising a source of a model and a data set, the given cloud platform communicating with a multi-cloud computing environment comprising one or more of other cloud platforms, the given cloud platform:
storing the model and the data set as one or more local files;
maintaining a given decentralized file system node and a given decentralized message bus node of a decentralized architecture of the multi-cloud computing environment, the decentralized architecture comprising a file system and a message bus;
sharing the model and the data set with the one or more of the other cloud platforms via the given decentralized file system node;
sending one or more messages related to execution of the model against the data set to the one or more of the other cloud platforms via the given decentralized message bus node, the one or more messages comprising instructions regarding execution of the model against the data set to enable execution across the one or more of the other cloud platforms;
enabling, via the given decentralized file system node, access to and receipt of first results of the execution of the model against the data set by a first execution cloud platform of the one or more of the other cloud platforms; and
enabling, via the given decentralized file system node, access to and receipt of second results of at least a subsequent execution of the model against the data set by a second execution cloud platform of the one or more of the other cloud platforms;
wherein the message bus comprises a distributed ledger system;
wherein the file system comprises a content address-based distributed file system; and
wherein the method is implemented via one or more processing devices each comprising a processor coupled to a memory.
2. The method of claim 1 , further comprising the given decentralized file system node of the given cloud platform receiving the model and the data set from the given cloud platform.
3. The method of claim 2 , further comprising the given decentralized file system node of the given cloud platform enabling access of the model and the data set to a first decentralized file system node of the first execution cloud platform of the one or more of the other cloud platforms to permit execution of the model against the data set by the first execution cloud platform.
4. The method of claim 3 , further comprising receiving by the given decentralized message bus node of the given cloud platform a message through a first decentralized message bus node of the first execution cloud platform indicating availability of the first results.
5. The method of claim 1 , wherein the model comprises a training model in the form of an analytic algorithm and wherein the data set against which the model is executed is a training data set.
6. The method of claim 3 , further comprising receiving by the given decentralized message bus node of the given cloud platform a message through a second decentralized message bus node of the second execution cloud platform indicating availability of the second results.
7. The method of claim 1 , wherein:
the given cloud platform comprises one or more private cloud platforms; and
each of the first execution cloud platform and the second execution cloud platform comprises one or more public cloud platforms.
8. An article of manufacture comprising a non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code when executed by at least one processing device causes said at least one processing device to:
in a given cloud platform comprising a source of a model and a data set, the given cloud platform communicating with a multi-cloud computing environment comprising one or more of other cloud platforms;
store the model and the data set as one or more local files;
maintain a given decentralized file system node and a given decentralized message bus node of a decentralized architecture of the multi-cloud computing environment, the decentralized architecture comprising a file system and a message bus;
share the model and the data set with the one or more of the other cloud platforms via the given decentralized file system node;
send one or more messages related to execution of the model against the data set to the one or more of the other cloud platforms via the given decentralized message bus node, the one or more messages comprising instructions regarding execution of the model against the data set to enable execution across the one or more of the other cloud platforms;
enable, via the given decentralized file system node, access to and receipt of first results of the execution of the model against the data set by a first execution cloud platform of the one or more of the other cloud platforms; and
enable, via the given decentralized file system node, access to and receipt of second results of at least a subsequent execution of the model against the data set by a second execution cloud platform of the one or more of the other cloud platforms;
wherein the message bus comprises a distributed ledger system; and
wherein the file system comprises a content address-based distributed file system.
9. The article of manufacture of claim 8 , further comprising the given decentralized file system node of the given cloud platform receiving the model and the data set from the given cloud platform.
10. The article of manufacture of claim 9 , further comprising the given decentralized file system node of the given cloud platform enabling access of the model and the data set to a first decentralized file system node of the first execution cloud platform of the one or more of the other cloud platforms to permit execution of the model against the data set by the first execution cloud platform.
11. The article of manufacture of claim 10 , further comprising receiving by the given decentralized message bus node of the given cloud platform a message through a first decentralized message bus node of the first execution cloud platform indicating availability of the first results.
12. The article of manufacture of claim 8 , wherein the model comprises a training model in the form of an analytic algorithm and wherein the data set against which the model is executed is a training data set.
13. The article of manufacture of claim 11 , further comprising receiving by the given decentralized message bus node of the given cloud platform a message through a second decentralized message bus node of the second execution cloud platform indicating availability of the second results.
14. The article of manufacture of claim 8 , wherein:
the given cloud platform comprises one or more private cloud platforms; and
each of the first execution cloud platform and the second execution cloud platform comprises one or more public cloud platforms.
15. A system comprising:
in a given cloud platform comprising a source of a model and a data set, the given cloud platform communicating with a multi-cloud computing environment comprising one or more of other cloud platforms;
one or more processing devices configured to:
store the model and the data set as one or more local files;
maintain a given decentralized file system node and a given decentralized message bus node of a decentralized architecture of the multi-cloud computing environment, the decentralized architecture comprising a file system and a message bus;
share the model and the data set with the one or more of the other cloud platforms via the given decentralized file system node;
send one or more messages related to execution of the model against the data set to the one or more of the other cloud platforms via the given decentralized message bus node, the one or more messages comprising instructions regarding execution of the model against the data set to enable execution across the one or more of the other cloud platforms;
enable, via the given decentralized file system node, access to and receipt of first results of the execution of the model against the data set by a first execution cloud platform of the one or more of the other cloud platforms; and
enable, via the given decentralized file system node, access to and receipt of second results of at least a subsequent execution of the model against the data set by a second execution cloud platform of the one or more of the other cloud platforms;
wherein the message bus comprises a distributed ledger system; and
wherein the file system comprises a content address-based distributed file system.
16. The system of claim 15 , further comprising the given decentralized file system node of the given cloud platform receiving the model and the data set from the given cloud platform.
17. The system of claim 16 , further comprising the given decentralized file system node of the given cloud platform enabling access of the model and the data set to a first decentralized file system node of the first execution cloud platform of the one or more of the other cloud platforms to permit execution of the model against the data set by the first execution cloud platform.
18. The system of claim 17 , further comprising receiving by the given decentralized message bus node of the given cloud platform a message through a first decentralized message bus node of the first execution cloud platform indicating availability of the first results.
19. The system of claim 18 , further comprising receiving by the given decentralized message bus node of the given cloud platform a message through a second decentralized message bus node of the second execution cloud platform indicating availability of the second results.
20. The system of claim 15 , wherein:
the given cloud platform comprises one or more private cloud platforms; and
each of the first execution cloud platform and the second execution cloud platform comprises one or more public cloud platforms.