Event-driven programming model based on asynchronous, massively parallel dataflow processes for highly-scalable distributed applications
An example method comprises receiving one or more published events by an event hook application program interface (API) from one or more client applications, passing a model to a web server configured to generate web containers in concurrent threads, receiving, by any number of worker nodes, each web container, each of the worker nodes including a system agent program for dynamically assigned functions, the web containers being provided to the any number of worker nodes for logical isolation of system agent execution in memory, and performing the dynamically assigned functions by the system agent program in a blackboard memory, the blackboard memory being a shared memory with non-blocking reads and writes and performing functionality, the dynamically assigned functions being executed in parallel and at least two of the dynamically assigned functions sharing context between inter-dependent processes.
1. A system for implementing complex distributed events in a cloud-native architecture, the system comprising:
a memory; and
at least one processor coupled to the memory and configured to:
receive one or more published events by an event hook application program interface (API) from one or more client applications;
generate a plurality of web containers in concurrent threads;
dispatch, to any number of worker nodes, each web container in the plurality of web containers, wherein each of the any number of worker nodes comprises a system agent program for dynamically assigned functions, and wherein each of the plurality of web containers logically isolates an execution of the system agent program in memory; and
perform the dynamically assigned functions by the system agent program in a blackboard memory, the blackboard memory being a shared memory with non-blocking reads and writes and performing functionality, the dynamically assigned functions being executed in parallel and at least two of the dynamically assigned functions sharing context between inter-dependent processes.
2. The system of claim 1 , wherein the any number of worker nodes are dynamically assigned the plurality of web containers based at least in part on the one or more published events from the one or more client applications.
3. The system of claim 2 , wherein the dynamically assigning the plurality of web containers enables the system agent program to execute utilizing at least one of the plurality of web containers to act as an event broker.
4. The system of claim 1 , the at least one processor further configured to:
expose a listener as an event hook interface to dispatch schedule-free, non-blocking, concurrent, multi-threaded, logically isolated web containers for the any number of worker nodes.
5. The system of claim 1 , wherein the blackboard memory implements the shared memory as immutable, append-only, log style persistence which supports the non-blocking reads and writes.
6. The system of claim 1 , wherein the non-blocking reads and writes to the blackboard memory follow a Command Query Responsibility Segregation (CQRS) pattern.
7. The system of claim 6 , the at least one processor further configured to:
write to a database from the blackboard memory a stream of observable events for event sourcing to allow event chaining.
8. The system of claim 7 , wherein the event chaining enables processors to be modeled as dataflows.
9. The system of claim 8 , the at least one processor further configured to:
model system microflow as a dataflow process based at least in part on the blackboard memory, the CQRS pattern, event-sourcing, and event chaining models.
10. The system of claim 9 , wherein the system agent program utilizes one or more dataflows to decompose complex event processing into a set of discrete tasks as part of the microflow.
11. The system of claim 1 , the at least one processor further configured to:
process a system language that supports Common Object Model based on Directed Acyclic Graphs (DAGs) implementation as isomorphism.
12. The system of claim 11 , the at least one processor further configured to:
provide a system runtime that supports Monadic Transformer support efficient DAG processing for implicitly complex objects.
13. A non-transitory computer readable medium comprising executable instructions, the instructions being executable by a processor to perform operations for implementing complex distributed events in a cloud-native architecture, the operations comprising:
receiving one or more published events by an event hook application program interface (API) from one or more client applications;
generating a plurality of web containers in concurrent threads;
dispatching to any number of worker nodes, each web container in the plurality of web containers, wherein each of the any number of worker nodes comprises a system agent program for dynamically assigned functions, and wherein each of the plurality of web containers logically isolates an execution of the system agent program in memory; and
performing the dynamically assigned functions by the system agent program in a blackboard memory, the blackboard memory being a shared memory with non-blocking reads and writes and performing functionality, the dynamically assigned functions being executed in parallel and at least two of the dynamically assigned functions sharing context between inter-dependent processes.
14. The non-transitory computer readable medium of claim 13 , wherein the any number of worker nodes are dynamically assigned the plurality of web containers based at least in part on the one or more published events from the one or more client applications.
15. The non-transitory computer readable medium of claim 14 , wherein the dynamically assigning the plurality of web containers enables the system agent program to execute utilizing at least one of the plurality of web containers to act as an event broker.
16. The non-transitory computer readable medium of claim 13 , the operations further comprising:
exposing a listener as an event hook interface to dispatch schedule-free, non-blocking, concurrent, multi-threaded, logically isolated web containers for the any number of worker nodes.
17. The non-transitory computer readable medium of claim 13 , wherein the blackboard memory implements the shared memory as immutable, append-only, log style persistence which supports the non-blocking reads and writes.
18. The non-transitory computer readable medium of claim 13 , wherein the non-blocking reads and writes to the blackboard memory follow a Command Query Responsibility Segregation (CQRS) pattern.
19. The non-transitory computer readable medium of claim 18 , the operations further comprising:
writing to a database from the blackboard memory a stream of observable events for event sourcing to allow event chaining.
20. The non-transitory computer readable medium of claim 19 , wherein the event chaining enables processors to be modeled as dataflows.
21. The non-transitory computer readable medium of claim 20 , the operations further comprising modeling system microflow as a dataflow process based at least in part on the blackboard memory, the CQRS pattern, event-sourcing, and event chaining models.
22. The non-transitory computer readable medium of claim 21 , wherein the system agent program utilizes one or more dataflows to decompose complex event processing into a set of discrete tasks as part of the microflow.
23. The non-transitory computer readable medium of claim 13 , the operations further comprising:
processing a system language that supports Common Object Model based on Directed Acyclic Graphs (DAGs) implementation as isomorphism.
24. The non-transitory computer readable medium of claim 23 , the operations further comprising:
providing a system runtime that supports Monadic Transformer support efficient DAG processing for implicitly complex objects.
25. A method for implementing complex distributed events in a cloud-native architecture, the method comprising:
receiving one or more published events by an event hook application program interface (API) from one or more client applications;
generating a plurality of web containers in concurrent threads;
dispatching to any number of worker nodes, each web container in the plurality of web containers, wherein each of the any number of worker nodes comprises a system agent program for dynamically assigned functions, and wherein each of the plurality of web containers logically isolates an execution of the system agent program in memory; and
performing the dynamically assigned functions by the system agent program in a blackboard memory, the blackboard memory being a shared memory with non-blocking reads and writes and performing functionality, the dynamically assigned functions being executed in parallel and at least two of the dynamically assigned functions sharing context between inter-dependent processes.