Brokerless reliable totally ordered many-to-many interprocess communication on a single node that uses shared memory and multicast
Examples described herein include systems and methods for brokerless reliable totally ordered many-to-many inter-process communication on a single node. A messaging protocol is provided that utilizes shared memory for one of the control plane and data plane, and multicast for the other plane. Readers and writers can store either control messages or message data in the shared memory, including in a ring buffer. Write access to portions of the shared memory can be controlled by a robust futex, which includes a locking mechanism that is crash recoverable. In general, the writers and readers can control the pace of communications and the crash of any process does not crash the overall messaging on the node.
1. A method for messaging on a physical device, comprising:
providing, at the physical device, a messaging protocol that includes a control plane and a data plane, wherein the control plane uses shared memory and the data plane uses user datagram protocol (“UDP”) multicast;
writing a message by a writer, including:
increasing a sequence identifier in shared memory, wherein the sequence identifier orders messages being sent using the messaging protocol;
adding the message under the sequence identifier in a seed buffer; and
updating a message position field in shared memory to announce the presence of the message; and
requesting the message by a reader, including:
when the message position field differs from a prior copy of that field stored by the reader, writing the sequence identifier to a request buffer; and
detecting, by the writer, the sequence identifier in the request buffer; and
sending a datagram corresponding to the message to the reader using UDP multicast, wherein the message is retrieved from the seed buffer using the sequence identifier,
wherein the reader and writer are processes of one or more applications executing at the node.
2. The method of claim 1 , wherein the writer resets the request buffer value so that the message is not sent to the reader twice.
3. The method of claim 1 , wherein the reader misses the message and requests the message again by rewriting the sequence identifier in the request buffer.
4. The method of claim 1 , wherein the request buffer is a ring buffer of sequence identifiers.
5. The method of claim 1 , wherein:
the reader receives the datagram out of order in a cache of received messages;
the order of cached received messages is corrected; and
the cached messages are read in the corrected order.
6. The method of claim 1 , wherein the reader notifies the writer regarding changes to the request buffer and the writer notifies the reader regarding changes to the message position field.
7. The method of claim 1 , wherein the reader includes an in-process cache to store received datagrams, and wherein the reader sends the cached datagrams to an application process in an order determined by sequence identifiers of the cached datagrams.
8. A non-transitory, computer-readable medium containing instructions that, when executed by a processor of a node, cause the node to perform stages comprising:
providing a messaging protocol that includes a control plane and a data plane, wherein the control plane uses shared memory and the data plane uses user datagram protocol (“UDP”) multicast;
writing a message by a writer, including:
increasing a sequence identifier in the shared memory, wherein the sequence identifier orders messages being sent using the messaging protocol;
adding the message under the sequence identifier in a seed buffer; and
updating a message position field in the shared memory to announce the presence of the message; and
requesting the message by a reader when the message position field differs from a prior copy of that field stored by the reader, wherein the reader inserts the sequence identifier into a request buffer in the shared memory; and
detecting, by the writer, the sequence identifier in the request buffer; and
sending a datagram corresponding to the message to the reader using UDP multicast, wherein the message is retrieved from the seed buffer using the sequence identifier,
wherein the reader and writer are processes of one or more applications executing at the node.
9. The non-transitory, computer-readable medium of claim 8 , wherein the writer resets the request buffer value so that the message is not sent to the reader twice.
10. The non-transitory, computer-readable medium of claim 8 , wherein the reader misses the message and requests the message again by rewriting the sequence identifier in the request buffer.
11. The non-transitory, computer-readable medium of claim 8 , wherein the request buffer is a ring buffer, and wherein multiple readers make message requests by writing to the ring buffer, and wherein multiple writers read the message requests from the ring buffer.
12. The non-transitory, computer-readable medium of claim 8 , wherein:
the reader receives the datagram out of order in a cache of received messages;
the order of cached received messages is corrected; and
the cached messages are read in the corrected order.
13. The non-transitory, computer-readable medium of claim 8 , wherein the reader notifies the writer regarding changes to the request buffer.
14. The non-transitory, computer-readable medium of claim 8 , wherein the reader includes an in-process cache to store received datagrams, and wherein the reader sends the cached datagrams to an upper layer of a first application of the one or more applications in an order determined by sequence identifiers of the cached datagrams.
15. A system for inter-process communication at a single node, the system comprising:
a non-transitory, computer-readable medium containing instructions;
a device that executes the instructions to perform stages comprising:
providing a messaging protocol that includes a control plane and a data plane, wherein the control plane uses shared memory and the data plane uses user datagram protocol (“UDP”) multicast;
writing a message by a writer, including:
increasing a sequence identifier in the shared memory, wherein the sequence identifier orders messages being sent using the messaging protocol;
adding the message under the sequence identifier in a seed buffer; and
updating a message position field in the shared memory to announce the presence of the message; and
requesting the message by a reader when the message position field differs from a prior copy of that field stored by the reader, wherein the reader inserts the sequence identifier into a request buffer in the shared memory; and
detecting, by the writer, the sequence identifier in the request buffer; and
sending a datagram corresponding to the message to the reader using UDP multicast, wherein the message is retrieved from the seed buffer using the sequence identifier,
wherein the reader and writer are processes of one or more applications executing at the node.
16. The system of claim 15 , wherein the writer resets the request buffer value so that the message is not sent to the reader twice.
17. The system of claim 15 , wherein the reader misses the message and requests the message again by rewriting the sequence identifier in the request buffer.
18. The system of claim 15 , wherein the request buffer is a ring buffer, and wherein multiple readers make message requests by writing to the ring buffer, and wherein multiple writers read the message requests from the ring buffer.
19. The system of claim 15 , wherein:
the reader receives the datagram out of order in a cache of received messages;
the order of cached received messages is corrected; and
the cached messages are read in the corrected order.
20. The system of claim 15 , wherein the reader includes an in-process cache to store received datagrams, and wherein the reader sends the cached datagrams to an upper layer of the one or more applications in an order determined by sequence identifiers of the cached datagrams.