Real-time streaming data ingestion into database tables
A streaming ingest platform can improve latency and expense issues related to uploading data into a cloud data system. The streaming ingest platform can organize the data to be ingested into per-table chunks and per-account blobs. This data may be committed and may be made available for query processing before it is ingested into the target source tables. This significantly improves latency issues. The streaming ingest platform can also accommodate uploading data from various sources with different processing and communication capabilities, such as Internet of Things (IOT) devices.
1. A method comprising:
receiving data from a client via one or more channels for ingestion into one or more source tables in a data system, the received data including sequencing information;
buffering the received data into per-table sets and into per-account groups in a buffer;
in response to determining that a threshold of the buffer has been exceeded, storing the received data in a storage in a first format;
generating per-table set registration requests for the received data;
deduping and validating the received data using the sequencing information;
based on the registration requests, committing, by a processor, the received data stored in the storage and making the received data in the first format accessible for query processing before the received data is ingested into the one or more source tables; and
ingesting the received data into the one or more source tables in a second format.
2. The method of claim 1 , wherein data in each set belongs to a single source table.
3. The method of claim 2 , wherein data in each group belongs to a single account.
4. The method of claim 1 , wherein ordering of the received data is maintained based on the sequencing information.
5. The method of claim 1 , further comprising:
writing the received data to a metadata store; and
generating a hybrid table for query processing, the hybrid table including the committed data in the first format and data from the one or more source tables in the second format.
6. The method of claim 5 , further comprising:
for query processing:
converting the committed data from the first format into a common format;
converting the data from the one or more source tables into the common format;
joining the committed data in the common format and the data from the one or more source tables in the common format to generate joined data; and
executing a query based on the joined data.
7. The method of claim 6 , further comprising:
retrieving expression properties of the received data; and
pruning the received data based on the expression properties and the query.
8. A machine-storage medium embodying instructions that, when executed by a machine, cause the machine to perform operations comprising:
receiving data from a client via one or more channels for ingestion into one or more source tables in a data system, the received data including sequencing information;
buffering the received data into per-table sets and into per-account groups in a buffer;
in response to determining that a threshold of the buffer has been exceeded, storing the received data in a storage in a first format;
generating per-table set registration requests;
deduping and validating the received data using the sequencing information;
based on the registration requests, committing, the received data stored in the storage and making the received data in the first format accessible for query processing before the received data is ingested into the one or more source tables; and
ingesting the received data into the one or more source tables in a second format.
9. The machine-storage medium of claim 8 , wherein data in each set belongs to a single source table.
10. The machine-storage medium of claim 9 , wherein per-table sets are organized into per-account groups, data in each group belonging to a single account.
11. The machine-storage medium of claim 8 , wherein ordering of the received data is maintained based on the sequencing information.
12. The machine-storage medium of claim 8 , further comprising:
writing the received data to a metadata store; and
generating a hybrid table for query processing, the hybrid table including the committed data in the first format and data from the one or more source tables in the second format.
13. The machine-storage medium of claim 12 , further comprising:
for query processing:
converting the committed data from the first format into a common format;
converting the data from the one or more source tables into the common format;
joining the committed data in the common format and the data from the one or more source tables in the common format to generate joined data; and
executing a query based on the joined data.
14. The machine-storage medium of claim 13 , further comprising:
retrieving expression properties of the received data; and
pruning the received data based on the expression properties and the query.
15. A system comprising:
at least one hardware processor; and
at least one memory storing instructions that, when executed by the at least one hardware processor, cause the at least one hardware processor to perform operations comprising:
receiving data from a client via one or more channels for ingestion into one or more source tables in a data system, the received data including sequencing information;
buffering the received data into per-table sets and into per-account groups in a buffer;
in response to determining that a threshold of the buffer has been exceeded, storing the received data in a storage in a first format;
generating per-table set registration requests;
deduping and validating the received data using the sequencing information;
based on the registration requests, committing the received data stored in the storage and making the received data in the first format accessible for query processing before the received data is ingested into the one or more source tables; and
ingesting the received data into the one or more source tables in a second format.
16. The system of claim 15 , wherein data in each set belongs to a single source table.
17. The system of claim 16 , wherein data in each group belongs to a single account.
18. The system of claim 15 , wherein ordering of the received data is maintained based on the sequencing information.
19. The system of claim 15 , the operations further comprising:
writing the received data to a metadata store; and
generating a hybrid table for query processing, the hybrid table including the committed data in the first format and data from the one or more source tables in the second format.
20. The system of claim 19 , the operations further comprising:
for query processing:
converting the committed data from the first format into a common format;
converting the data from the one or more source tables into the common format;
joining the committed data in the common format and the data from the one or more source tables in the common format to generate joined data; and
executing a query based on the joined data.
21. The system of claim 20 , the operations further comprising:
retrieving expression properties of the received data; and
pruning the received data based on the expression properties and the query.