Publish-subscribe framework for application execution
The described technology relates to a publish-subscribe message framework in which an application, decomposed to a plurality of processing stages, is run by executing respective processing stages of the application asynchronously and simultaneously with each other. Communications between the respective processing stages may exclusively be in accordance with the publish-subscribe execution model. The described publish-subscribe framework provides for processing stages to be executed in a multi-process and/or multi-threaded manner while also enabling the distribution of the processing stages to respective processing resources in a multi-processor/multi-core processing environment. An example electronic exchange application and a corresponding example exchange gateway application are described.
1. A distributed processing system, comprising:
at least one memory; and
a plurality of processing resources,
wherein the at least one memory and the plurality of processing resources are configured to perform operations comprising:
receiving processing logic instructions;
decomposing, in accordance with one or more configuration parameters, the processing logic instructions to obtain a plurality of pipeline stages of a processing pipeline;
executing, on each of the processing resources, one or more pipeline stages of the plurality of pipeline stages, wherein communication between respective pipeline stages of the plurality of pipeline stages occur via publish-subscribe messages;
registering each of the plurality of pipeline stages as a publisher and/or subscriber in a session for processing a received data message;
processing the received data message in one or more respective registered pipeline stages of the registered plurality of the pipeline stages of the session to generate a processed message, wherein said processing the received data message includes messages being exchanged between respective pairs of publishers and subscribers among the registered publishers and the registered subscribers via at least one message queue; and
outputting the processed data message,
wherein the at least one memory and the plurality of processing resources are further configured to dynamically determine a number of instances of at least one of the plurality of pipeline stages for processing the received data message.
2. The distributed processing system according to claim 1 , wherein the at least one memory and the plurality of processing resources are further configured to:
instantiate a control session;
subscribe each of the pipeline stages to the control session; and
provide for signaling one or more predetermined events to the plurality of pipeline stages by publishing to the control session.
3. The distributed processing system according to claim 1 , wherein the at least one memory and the plurality of processing resources are further configured to determine an assignment of the plurality of pipeline stages to the plurality of processing resources.
4. The distributed processing system according to claim 3 , wherein the at least one memory and the plurality of processing resources are further configured to determine the assignment based on an ingress message interface and an egress message interface on each of the pipeline stages.
5. The distributed processing system according to claim 1 , wherein the registering each of the plurality of pipeline stages as a publisher and/or subscriber in a session for processing a received data message comprises each of the plurality of pipeline stages performing at least one of publishing a topic to the session or subscribing to a topic on the session.
6. The distributed processing system according to claim 5 , wherein the registering each of the plurality of pipeline stages as a publisher and/or subscriber in a session for processing a received data message further comprises at least one of the plurality of pipeline stages performing the subscribing to a topic on the session in response to a received message notifying of the topic.
7. The distributed processing system according to claim 1 , wherein the at least one memory and the plurality of processing resources are further configured to perform said processing the received data message by accessing a transaction processing engine with respect to the data message.
8. The distributed processing system according to claim 7 , wherein the at least one memory and the plurality of processing resources are further configured to perform said processing the received data message in a plurality of the pipeline stages by said accessing the transaction processing engine in parallel for a first transaction request and another transaction request.
9. A method of processing a data message in a distributed processing system having at least one memory and a plurality of processing resources, the method comprising:
receiving processing logic instructions;
decomposing, in accordance with one or more configuration parameters, the processing logic instructions to obtain a plurality of pipeline stages of a processing pipeline;
executing, on each of the processing resources, one or more pipeline processing stages of the plurality of pipeline stages, wherein communication between respective pipeline stages of the plurality of pipeline stages occur via publish-subscribe messages;
registering each of the plurality of pipeline stages as a publisher and/or subscriber in a session for processing a received data message;
processing the received data message in one or more respective registered pipeline stages of the registered plurality of the pipeline stages of the session to generate output processed message, wherein said processing the received data message includes messages being exchanged between respective pairs of publishers and subscribers among the registered publishers and the registered subscribers via at least one message queue; and
outputting the processed data message,
wherein the method further comprises dynamically determining a number of instances of at least one of the plurality of pipeline stages for processing the received data message.
10. The method according to claim 9 , further comprising:
instantiating a control session;
subscribing each of the pipeline stages to the control session; and
providing for signaling one or more predetermined events to the plurality of pipeline stages by publishing to the control session.
11. The method according to claim 9 , wherein the registering each of the plurality of pipeline stages as a publisher and/or subscriber in a session for processing a received data message comprises each of the plurality of pipeline stages performing at least one of publishing a topic to the session or subscribing to a topic on the session.
12. The method according to claim 11 , wherein the registering each of the plurality of pipeline stages as a publisher and/or subscriber in a session for processing a received data message further comprises at least one of the plurality of pipeline stages performing the subscribing to a topic on the session in response to a received message notifying of the topic.
13. The method according to claim 9 , further comprising performing said processing the received data message by accessing a transaction processing engine with respect to the data message.
14. The method according to claim 13 , further comprising performing said processing the received data message in a plurality of the pipeline stages by said accessing the transaction processing engine in parallel for a first transaction request and another transaction request.
15. A non-transitory computer-readable storage medium storing instructions which, when executed by a plurality of processing resources of a distributed processing system that comprises at least one memory and the plurality of processing resources, causes the distributed processing system to perform operations comprising:
receiving a processing logic instructions;
decomposing, in accordance with one or more configuration parameters, the processing logic instructions to obtain a plurality of pipeline stages of a processing pipeline;
executing, on each of the processing resources, one or more pipeline processing stages of the plurality of pipeline stages, wherein communication between respective pipeline stages of the plurality of pipeline stages occurring via publish-subscribe messages;
registering each of the plurality of pipeline stages as a publisher and/or subscriber in a session for processing a received data message;
processing the received data message in one or more respective registered pipeline stages of the registered plurality of the pipeline stages of the session to generate a processed message, wherein said processing the received data message includes messages being exchanged between respective pairs of publishers and subscribers among the registered publishers and the registered subscribers via at least one message queue; and
outputting the processed data message,
wherein the operations further comprise dynamically determining a number of instances of at least one of the plurality of pipeline stages for processing the received data message.
16. The non-transitory computer-readable storage medium according to claim 15 , wherein the stored instructions which, when executed by the plurality of processing resources, causes the distributed processing system to perform operations further comprising:
instantiating a control session;
subscribing each of the pipeline stages to the control session; and
providing for signaling one or more predetermined events to the plurality of pipeline stages by publishing to the control session.
17. The non-transitory computer-readable storage medium according to claim 15 , wherein the registering each of the plurality of pipeline stages as a publisher and/or subscriber in a session for processing a received data message comprises each of the plurality of pipeline stages performing at least one of publishing a topic to the session or subscribing to a topic on the session.
18. The non-transitory computer-readable storage medium according to claim 15 , wherein the stored instructions which, when executed by the plurality of processing resources, causes the distributed processing system to perform further operations comprising, performing said processing the received data message by accessing a transaction engine with respect to the data message.
19. The non-transitory computer-readable storage medium according to claim 18 , wherein the stored instructions which, when executed by the plurality of processing resources, causes the distributed processing system to perform further operations comprising, performing said processing the received data message in a plurality of the pipeline stages by said accessing the transaction engine in parallel for a first transaction request and another transaction request.