Event-driven programming model based on asynchronous, massively parallel dataflow processes for highly-scalable distributed applications
An example method comprises receiving one or more published events by an event hook application program interface (API) from one or more client applications, passing a model to a web server configured to generate web containers in concurrent threads, receiving, by any number of worker nodes, each web container, each of the worker nodes including a system agent program for dynamically assigned functions, the web containers being provided to the any number of worker nodes for logical isolation of system agent execution in memory, and performing the dynamically assigned functions by the system agent program in a blackboard memory, the blackboard memory being a shared memory with non-blocking reads and writes and performing functionality, the dynamically assigned functions being executed in parallel and at least two of the dynamically assigned functions sharing context between inter-dependent processes.
1. A system for implementing complex distributed events in a cloud-native architecture, the system comprising:
one or more processors; and
system memory storing instructions that, when executed by the one or more processors, cause the system to perform:
receiving one or more published events by an event hook application program interface (API) from one or more client applications;
passing a model to a web server configured to generate web containers in concurrent threads;
receiving, by any number of worker nodes, each web container, each of the worker nodes including a corresponding system agent program for corresponding dynamically assigned functions, the web containers being provided to the any number of worker nodes for logical isolation of each of the corresponding system agent programs' execution in the system memory; and
performing the corresponding dynamically assigned functions by each of the corresponding system agent programs in at least one blackboard memory, the at least one blackboard memory being at least one shared memory with non-blocking reads and writes and performing functionality, the corresponding dynamically assigned functions being executed in parallel, and at least two of the corresponding dynamically assigned functions sharing context between inter-dependent processes.
2. The system of claim 1 , wherein the any number of worker nodes are dynamically assigned the web containers based at least in part on the one or more published events from the one or more client applications.
3. The system of claim 2 , wherein the dynamically assigning the web containers enables the system agent programs to execute utilizing at least one of the web containers to act as an event broker.
4. The system of claim 1 , wherein the web server is a listener exposed as an event hook interface to dispatch schedule-free, non-blocking, concurrent, multi-threaded, logically isolated web containers for the any number of worker nodes.
5. The system of claim 1 , wherein the at least one blackboard memory implements the at least one shared memory as immutable, append-only, log style persistence which supports the non-blocking reads and writes.
6. The system of claim 1 , wherein the reads and writes to the at least one blackboard memory follow a Command Query Responsibility Segregation (CQRS) pattern.
7. The system of claim 6 , wherein the instructions, when executed by the one or more processors, cause the system to further perform writing to a database from the at least one blackboard memory, the writing to the database being as a stream of observable events for event sourcing, thereby allowing event chaining.
8. The system of claim 7 , wherein the event chaining enables processes to be modeled as dataflows.
9. The system of claim 8 , wherein the instructions, when executed by the one or more processors, cause the system to further perform modeling system microflow as a dataflow process based at least in part on the at least one blackboard memory, the CQRS pattern, the event-sourcing, and event chaining models.
10. The system of claim 9 , wherein the system agent programs utilize one or more dataflows to decompose complex event processing into a set of discrete tasks as part of the microflow.
11. The system of claim 1 , wherein the web server includes a system language that supports Common Object Model based on Directed Acyclic Graphs (DAGs) implementation as isomorphism.
12. The system of claim 11 , wherein the web server includes a system runtime that supports Monadic Transformer support efficient DAG processing for implicitly complex objects.
13. A non-transitory computer readable medium comprising executable instructions, the instructions being executable by a processor to perform a method for implementing complex distributed events in a cloud-native architecture, the method comprising:
receiving one or more published events by an event hook application program interface (API) from one or more client applications;
passing a model to a web server configured to generate web containers in concurrent threads;
receiving, by any number of worker nodes, each web container, each of the worker nodes including a corresponding system agent program for corresponding dynamically assigned functions, the web containers being provided to the any number of worker nodes for logical isolation of each of the corresponding system agent programs' execution in system memory; and
performing the corresponding dynamically assigned functions by each of the corresponding system agent programs in at least one blackboard memory, the at least one blackboard memory being at least one shared memory with non-blocking reads and writes and performing functionality, the corresponding dynamically assigned functions being executed in parallel, and at least two of the corresponding dynamically assigned functions sharing context between inter-dependent processes.
14. The non-transitory computer readable medium of claim 13 , wherein the any number of worker nodes are dynamically assigned the web containers based at least in part on the one or more published events from the one or more client applications.
15. The non-transitory computer readable medium of claim 14 , wherein the dynamically assigning the web containers enables the system agent programs to execute utilizing at least one of the web containers to act as an event broker.
16. The non-transitory computer readable medium of claim 13 , wherein the web server is a listener exposed as an event hook interface to dispatch schedule-free, non-blocking, concurrent, multi-threaded, logically isolated web containers for the any number of worker nodes.
17. The non-transitory computer readable medium of claim 13 , wherein the at least one blackboard memory implements the at least one shared memory as immutable, append-only, log style persistence which supports the non-blocking reads and writes.
18. The non-transitory computer readable medium of claim 13 , wherein the reads and writes to the at least one blackboard memory follow a Command Query Responsibility Segregation (CQRS) pattern.
19. The non-transitory computer readable medium of claim 18 , wherein the method further comprises writing to a database from the at least one blackboard memory, the writing to the database being as a stream of observable events for event sourcing, thereby allowing event chaining.
20. The non-transitory computer readable medium of claim 19 , wherein the event chaining enables processes to be modeled as dataflows.
21. The non-transitory computer readable medium of claim 20 , wherein the method further comprises modeling system microflow as a dataflow process based at least in part on the at least one blackboard memory, the CQRS pattern, the event-sourcing, and event chaining models.
22. The non-transitory computer readable medium of claim 21 , wherein the system agent programs utilize one or more dataflows to decompose complex event processing into a set of discrete tasks as part of the microflow.
23. The non-transitory computer readable medium of claim 13 , wherein the web server includes a system language that supports Common Object Model based on Directed Acyclic Graphs (DAGs) implementation as isomorphism.
24. The non-transitory computer readable medium of claim 23 , wherein the web server includes a system runtime that supports Monadic Transformer support efficient DAG processing for implicitly complex objects.
25. A method for implementing complex distributed events in a cloud-native architecture, the method comprising:
receiving one or more published events by an event hook application program interface (API) from one or more client applications;
passing a model to a web server configured to generate web containers in concurrent threads;
receiving, by any number of worker nodes, each web container, each of the worker nodes including a corresponding system agent program for corresponding dynamically assigned functions, the web containers being provided to the any number of worker nodes for logical isolation of each of the corresponding system agent programs' execution in system memory; and
performing the corresponding dynamically assigned functions by each of the corresponding system agent programs in at least one blackboard memory, the at least one blackboard memory being at least one shared memory with non-blocking reads and writes and performing functionality, the corresponding dynamically assigned functions being executed in parallel, and at least two of the corresponding dynamically assigned functions sharing context between inter-dependent processes.