Distributed system with fault tolerance and self-maintenance
A distributed system includes a plurality of compute nodes configured to process messages. The compute nodes each process messages corresponding an assigned value of a common parameter of the messages. The values are assigned to the compute nodes such that two or more compute nodes are available to process each message. The values can be assigned to the compute nodes in a grouping configuration or a striping configuration. The compute nodes also circulate one or more tokens among nodes, and perform a self-maintenance operation during a given state of possession of the token. During a self-maintenance operation, the values assigned to the compute node can be reassigned to other compute nodes to ensure processing of corresponding messages.
1. A distributed system comprising:
a plurality of compute nodes configured to process messages;
the plurality of compute nodes being configured to 1) circulate at least one token between or among at least two of the plurality of respective compute nodes, and 2) perform a self-maintenance operation during a given state of possession of the at least one token;
wherein the plurality of compute nodes further comprises: a first subset of at least two of the plurality of compute nodes, the first subset being configured to process messages corresponding to a first value of a common parameter of the messages; and
a second subset of at least two of the plurality of compute nodes being distinct from the first subset, the second subset being configured to process messages corresponding to a second value of the common parameter of the messages and to refrain from processing messages corresponding to the first value: wherein each compute node of the plurality of compute nodes is configured to refrain from performing the self-maintenance operation when not in the given state of possession of the at least one token.
2. The distributed system of claim 1 , wherein each of the compute nodes of the first subset is configured to process a message corresponding to the first value in parallel and independent from one another.
3. The distributed system of claim 1 , wherein, during the self-maintenance operation by one of the compute nodes of the first subset, at least one other compute node of the first subset is configured to process a message corresponding to the first value.
4. The distributed system of claim 1 , wherein the plurality of compute nodes is further configured to circulate the at least one token between or among the first and second subsets.
5. The distributed system of claim 1 , wherein the first subset is configured to circulate a first one of the at least one token between or among the plurality of compute nodes of the first subset, and wherein the second subset is configured to circulate a second one of the at least one token between or among the second subset.
6. A distributed system comprising:
a plurality of compute nodes configured to process messages;
the plurality of compute nodes being configured to 1) circulate at least one token between or among at least two of the plurality of respective compute nodes, and 2) perform a self-maintenance operation during a given state of possession of the at least one token;
wherein each of a plurality of values of a common parameter of the messages is assigned to at least two of the plurality of compute nodes, the at least two of the plurality of compute nodes circulating the at least one token between or among them;
wherein each compute node of the plurality of compute nodes is configured to refrain from performing the self-maintenance operation when not in the given state of possession of the at least one token.
7. The distributed system of claim 6 , wherein each of the plurality of compute nodes is assigned to process a respective subset of at least two of the plurality of values of the common parameter.
8. The distributed system of claim 7 , wherein each of the respective subsets differs from one another by at least one value of the common parameter.
9. The distributed system of claim 6 , wherein each of the plurality of compute nodes assigned to a given value of the common parameter is configured to process a message corresponding to the given value in parallel and independent from one another.
10. The distributed system of claim 7 , wherein, during a self-maintenance operation by a first compute node assigned to a first value of the common parameter, a second compute node is assigned to process messages corresponding to the first value in addition to the respective subset.
11. The distributed system of claim 6 , wherein each compute node of the plurality of compute nodes is configured to 1) receive the at least one token from a preceding one of the plurality of compute nodes; 2) perform the self-maintenance operation selectively based on a state of the compute node; and 3) forward the at least one token to a subsequent one of the plurality of compute nodes.
12. The distributed system of claim 6 , wherein the messages are associated with transactions of financial instruments, respective values of the common parameter of the messages each corresponding to a respective financial instrument or a transaction type.
13. The distributed system of claim 6 , further comprising:
at least one gateway configured to forward the messages to the plurality of compute nodes; and
at least one sequencer configured to sequence the messages.
14. The distributed system of claim 13 , wherein the plurality of compute nodes are further configured to forward a response to the at least one gateway after processing a message.
15. The distributed system of claim 13 , wherein the at least one gateway is further configured to transmit a message to a corresponding subset of the plurality of compute nodes as a function of a first value of the common parameter of the message.
16. The distributed system of claim 6 , wherein the self-maintenance operation includes at least one of 1) clearing data associated with at least one previous message processing operation, 2) moving data in memory, 3) adjusting layout of a memory, and 4) modifying a message queue.
17. The distributed system of claim 6 , wherein:
the at least one token includes a plurality of tokens, each of the plurality of tokens indicating a respective type of self-maintenance operation; and
the self-maintenance operation corresponds to the respective type.
18. The distributed system of claim 6 , wherein the plurality of compute nodes includes:
a first compute node configured to process messages corresponding to a first value and a second value of the common parameter of the messages;
a second compute node configured to process messages corresponding to the second value and a third value of the common parameter of the messages and to refrain from processing messages corresponding to the first value; and
a third compute node configured to process messages corresponding to the third value and a fourth value of the common parameter of the messages and to refrain from processing messages corresponding to the second value.
19. The distributed system of claim 18 , wherein, in response to an indication that the first compute node is not operational, at least one of the plurality of compute nodes is reconfigured to process messages corresponding to at least one of the first value and the second value.
20. A method of processing messages, comprising:
at a plurality of compute nodes, selectively processing messages based on a value of a common parameter of the messages; at the plurality of compute nodes, circulating at least one token between or among at least two of the plurality of respective compute nodes;
at the plurality of compute nodes, perform performing a self-maintenance operation during a given state of possession of the at least one token;
assigning each of the plurality of values of the common parameter to at least two of the plurality of compute nodes; assigning each of the plurality of compute nodes to process a respective subset of at least two of the plurality of values of the common parameter, wherein each of the respective subsets differs from one another by at least one value of the common parameter; and
at the plurality of compute nodes, refraining from performing the self-maintenance operation when not in the given state of possession of the at least one token.
21. A method of processing messages, comprising: at a plurality of compute nodes, selectively processing messages based on a value of a common parameter of the messages, at the plurality of compute nodes, circulating at least one token between or among at least two of the plurality of respective compute nodes, at the plurality of compute nodes, performing a self-maintenance operation during a given state of possession of the at least one token and refraining from performing the self-maintenance operation when not in the given state of possession of the at least one token; at a first subset of at least two of the plurality of compute nodes, processing messages corresponding to a first value of the common parameter of the messages; and at a second subset of at least two of the plurality of compute nodes being distinct from the first subset, processing messages corresponding to a second value of the common parameter of the messages and refraining from processing messages corresponding to the first value.
22. The method of claim 20 , further comprising, at each compute node of the plurality of compute nodes:
receiving the at least one token from a preceding one of the plurality of compute nodes;
performing the self-maintenance operation selectively based on a state of the compute node; and
forwarding the at least one token to a subsequent one of the plurality of compute nodes.
23. The method of claim 20 , wherein the messages are associated with transactions of financial instruments, the respective values of the common parameter each corresponding to a respective financial instrument or a transaction type.
24. The method of claim 20 , wherein the self-maintenance operation includes at least one of 1) clearing data associated with at least one previous message processing operation, 2) moving data in memory, 3) adjusting layout of a memory, and 4) modifying a message queue.
25. A method of processing messages, comprising:
at a first compute node: parsing a first message to determine a value of a common parameter of the message; selectively processing the message based on whether the value corresponds to an assigned common parameter associated with the first compute node; receiving a token from a second compute node; performing a self-maintenance operation during a given state of possession of the token and refraining from performing the self-maintenance operation when not in the given state of possession of the token; and sending the token to the second compute node or a third compute node; assigning each of a plurality of values of the common parameter to at least two of a plurality of compute nodes including the first, second and third compute nodes; and assigning each of the plurality of compute nodes to process a respective subset of at least two of the plurality of values of the common parameter, wherein each of the respective subsets differs from one another by at least one value of the common parameter.
26. The method of claim 25 , wherein the messages are associated with transactions of financial instruments, the respective values of the common parameter each corresponding to a respective financial instrument or a transaction type.