Brokerless reliable totally ordered many-to-many interprocess communication on a single node that uses shared memory and multicast
Examples described herein include systems and methods for brokerless reliable totally ordered many-to-many inter-process communication on a single node. A messaging protocol is provided that utilizes shared memory for one of the control plane and data plane, and multicast for the other plane. Readers and writers can store either control messages or message data in the shared memory, including in a ring buffer. Write access to portions of the shared memory can be controlled by a robust futex, which includes a locking mechanism that is crash recoverable. In general, the writers and readers can control the pace of communications and the crash of any process does not crash the overall messaging on the node.
1. A method for messaging on a physical device, comprising:
providing, at the physical device, a messaging protocol that includes a control plane and a data plane, wherein the data plane runs in shared memory and the control plane uses user datagram protocol (“UDP”) multicast;
writing, with a writer for an application, a message to the shared memory, wherein the writing includes:
requesting a robust futex for access to the shared memory, wherein a kernel marks the futex as acquired by the writer;
with access to the futex, writing the message to the shared memory;
after the message is written, updating a message position in the shared memory; and
releasing the futex;
sending, by the writer, a datagram using UDP multicast to notify readers of the existence of the message;
receiving, by a reader, the datagram; and
reading, by the reader, the shared memory up to the message position,
wherein the reader stores its stream position in-process, and wherein the reader reads the shared memory from the stream position to the message position based on receiving the datagram.
2. The method of claim 1 , wherein the shared memory includes a ring buffer comprising a stream of messages.
3. The method of claim 2 , wherein a plurality of writers including the writer can write to the ring buffer, and wherein the ring buffer is read only for a plurality of reader including the reader.
4. The method of claim 1 , wherein a stream position of the reader is stored in shared memory and read by the writer, and wherein the writer temporarily delays the writing when the stream position falls a threshold number of positions behind the message position.
5. The method of claim 1 , wherein a second writer can acquire the futex when the writer crashed prior to releasing the futex, wherein the marking indicates to the second writer that the writer failed unexpectedly.
6. The method of claim 1 , wherein prior to writing to the message position in the shared memory, the writer checks the stream position and stream positions of other readers to ensure the message position does not already reference an existing message not yet read by the reader or one of the other readers.
7. A non-transitory, computer-readable medium containing instructions that, when executed by a processor of a node, cause the node to perform stages comprising:
providing, at a physical device, a messaging protocol that includes a control plane and a data plane, wherein the data plane runs in shared memory and the control plane uses user datagram protocol (“UDP”) multicast;
writing, with a writer for an application, a message to the shared memory, wherein the writing includes:
requesting a robust futex for access the shared memory, wherein a kernel marks the futex as acquired by the writer;
with access to the futex, writing the message to the shared memory;
after the message is written, updating a message position in the shared memory; and
releasing the futex;
sending, by the writer, a datagram using UDP multicast to notify readers of the existence of the message;
receiving, by a reader, the datagram; and
reading, by the reader, the shared memory up to the message position,
wherein a stream position of the reader is stored in the shared memory and read by the writer, and wherein the writer temporarily delays the writing when the stream position falls a threshold number of positions behind the message position.
8. The non-transitory, computer-readable medium of claim 7 , wherein the shared memory includes a ring buffer comprising a stream of messages, wherein a plurality of writers including the writer can write to the ring buffer, and wherein the ring buffer is read only for a plurality of reader including the reader.
9. The non-transitory, computer-readable medium of claim 7 , wherein the datagram is sent after multiple new messages including the message are written to the shared memory.
10. The non-transitory, computer-readable medium of claim 7 , wherein the reader stores its stream position in-process, and wherein the reader reads the shared memory from the stream position to the message position based on receiving the datagram..
11. The non-transitory, computer-readable medium of claim 7 , wherein a second writer can acquire the futex when the writer crashed prior to releasing the futex, wherein the marking indicates to the second writer that the writer failed unexpectedly.
12. The non-transitory, computer-readable medium of claim 7 , wherein prior to writing to the message position in the shared memory, the writer checks the stream position and stream positions of other readers to ensure the message position does not already reference an existing message not yet read by the reader or one of the other readers.
13. A system for inter-process communication at a single node, the system comprising:
a non-transitory, computer-readable medium containing instructions;
a device that executes the instructions to perform stages comprising:
providing a messaging protocol that includes a control plane and a data plane, wherein the data plane runs in shared memory and the control plane uses user datagram protocol (“UDP”) multicast;
writing, with a writer, a message to the shared memory, wherein the writing includes:
requesting a robust futex for access the shared memory;
with access to the futex, writing the message to the shared memory;
after the message is written, updating a message position in the shared memory; and
releasing the futex;
sending, by the writer, a notification to notify readers of the existence of the message;
receiving, by a reader, the datagram; and
reading, by the reader, the shared memory up to the message position,
wherein the reader stores a stream position in-process to indicate a last read position by the reader, wherein the reader updates the shared memory to indicate the stream position, and wherein the writer pauses writing based on the stream position in the shared memory.
14. The system of claim 13 , wherein the shared memory includes a ring buffer comprising a stream of messages, and wherein the notification is a datagram sent using UDP multicast.
15. The system of claim 14 , wherein, when the message cannot fit at the message position, the writing includes adding a padding message at an end of the ring buffer with the message following the padding, wherein the reader ignores the padding message.
16. The system of claim 14 , wherein a plurality of writers including the writer can write to the ring buffer, and wherein the ring buffer is read only for a plurality of readers including the reader.
17. The system of claim 13 , wherein an operating system of the node marks the mutex to indicate a crash of the writer, wherein a second writer acquires the marked mutex without waiting for the writer to complete the writing stage, the marking indicating to the second writer that the crash occurred.