Distributed appending of transactions in data lakes

1. A computer-implemented method comprising:

receiving, at a first ingestion node of a plurality of ingestion nodes, as part of a transaction, a first message, the first message identifying a transaction identifier (ID), a first count of messages for the transaction, and a portion of data for the transaction;

persisting the data of the first message in temporary storage;

determining a second count of messages for the transaction for the first ingestion node;

based on at least the second count of messages, determining that the first ingestion node has received a complete set of messages for the transaction for the first ingestion node; and

transmitting, by the first ingestion node, to a coordinator, a metadata write request, the metadata write request identifying the transaction ID, the first count of messages, and the second count of messages, and including a self-describing reference to persisted data of the set of messages for the transaction for the first ingestion node, wherein the self-describing reference identifies the first ingestion node, location information of the persisted data, and a range of the first data.

2. The computer-implemented method of claim 1 , further comprising:

identifying, by the coordinator, on an ongoing basis, a transactionally complete sub-set of data being processed by a set of ingestion nodes; and

making the transactionally complete sub-set of data available for reading by a reader.

3. The computer-implemented method of claim 1 , further comprising:

identifying, by the coordinator, metadata write requests from a set of ingestion nodes of the plurality of ingestion nodes, the metadata write requests each including the transaction ID and a reference to persisted data, the set of ingestion nodes including the first ingestion node;

determining that the transaction is complete;

consolidating the references of the metadata write requests into a composite reference; and

persisting the composite reference.

4. The computer-implemented method of claim 1 , further comprising:

transmitting, by the coordinator, a flush advisory to the first ingestion node, wherein the flush advisory comprises a trigger for persisting the data of the first message in a permanent storage.

5. The computer-implemented method of claim 1 , further comprising:

persisting the data of the first message in a permanent storage; and

identifying, to the coordinator, location information and a range for the persisted data of the first message.

6. The computer-implemented method of claim 1 , further comprising:

partitioning messages for the transaction among the plurality of ingestion nodes, wherein the partitioning comprises assigning the first message to the first ingestion node.

7. The computer-implemented method of claim 1 , wherein the first message further identifies the second count of messages.

8. The computer-implemented method of claim 1 , further comprising:

receiving, by the coordinator, the metadata write request from the first ingestion node; and

receiving, by the coordinator, metadata write requests from others of the set of ingestion nodes.

9. A computer system comprising:

a processor; and

a non-transitory computer readable medium having stored thereon program code executable by the processor, the program code causing the processor to:

receive, at a first ingestion node of a plurality of ingestion nodes, as part of a transaction, a first message, the first message identifying a transaction identifier (ID), a first count of messages for the transaction, and a portion of data for the transaction;

persist the data of the first message in temporary storage;

determine a second count of messages for the transaction for the first ingestion node;

based on at least the second count of messages, determine that the first ingestion node has received a complete set of messages for the transaction for the first ingestion node; and

transmit, by the first ingestion node, to a coordinator, a metadata write request, the metadata write request identifying the transaction ID, the first count of messages, and the second count of messages, and including a self-describing reference to persisted data of the set of messages for the transaction for the first ingestion node, wherein the self-describing reference identifies the first ingestion node, location information of the persisted data, and a range of the first data.

10. The computer system of claim 9 , wherein the program code is further operative to:

identify, by the coordinator, metadata write requests from a set of ingestion nodes of the plurality of ingestion nodes, the metadata write requests each including the transaction ID and a reference to persisted data, the set of ingestion nodes including the first ingestion node;

determine that the transaction is complete;

consolidate the references of the metadata write requests into a composite reference; and

persist the composite reference.

11. The computer system of claim 9 , wherein the program code is further operative to:

transmit, by the coordinator, a flush advisory to the first ingestion node, wherein the flush advisory comprises a trigger for persisting the data of the first message in a permanent storage.

12. The computer system of claim 9 , wherein the program code is further operative to:

persist the data of the first message in a permanent storage; and

identify, to the coordinator, location information and a range for the persisted data of the first message.

13. The computer system of claim 9 , wherein the program code is further operative to:

partition messages for the transaction among the plurality of ingestion nodes, wherein the partitioning comprises assigning the first message to the first ingestion node.

14. The computer system of claim 9 , wherein the program code is further operative to:

receive, by the coordinator, the metadata write request from the first ingestion node; and

receive, by the coordinator, metadata write requests from others of the set of ingestion nodes.

15. A non-transitory computer storage medium having stored thereon program code executable by a processor, the program code embodying a method comprising:

receiving, at a first ingestion node of a plurality of ingestion nodes, as part of a transaction, a first message, the first message identifying a transaction identifier (ID), a first count of messages for the transaction, and a portion of data for the transaction;

persisting the data of the first message in temporary storage;

determining a second count of messages for the transaction for the first ingestion node;

based on at least the second count of messages, determining that the first ingestion node has received a complete set of messages for the transaction for the first ingestion node; and

transmitting, by the first ingestion node, to a coordinator, a metadata write request, the metadata write request identifying the transaction ID, the first count of messages, and the second count of messages, and including a self-describing reference to persisted data of the set of messages for the transaction for the first ingestion node, wherein the self-describing reference identifies the first ingestion node, location information of the persisted data, and a range of the first data.

16. The computer storage medium of claim 15 , wherein the program code method further comprises:

identifying, by the coordinator, metadata write requests from a set of ingestion nodes of the plurality of ingestion nodes, the metadata write requests each including the transaction ID and a reference to persisted data, the set of ingestion nodes including the first ingestion node;

determining that the transaction is complete;

consolidating the references of the metadata write requests into a composite reference; and

persisting the composite reference.

17. The computer storage medium of claim 16 , wherein the program code method further comprises:

prior to persisting the composite reference, routing a first read request for the data of the first message to the ingestion node; and

after persisting the composite reference, not routing a second read request for the data of the first message to the ingestion node.

18. The computer storage medium of claim 15 , wherein the program code method further comprises:

persisting the data of the first message in a permanent storage; and

identifying, to the coordinator, location information and a range for the persisted data of the first message.

19. The computer storage medium of claim 15 , wherein the program code method further comprises:

partitioning messages for the transaction among the plurality of ingestion nodes, wherein the partitioning comprises assigning the first message to the first ingestion node.

20. The computer storage medium of claim 15 , wherein the first message further identifies the second count of messages.