Real-time streaming data ingestion into database tables
A streaming ingest platform can improve latency and expense issues related to uploading data into a cloud data system. The streaming ingest platform can organize the data to be ingested into per-table chunks and per-account blobs. This data may be committed and may be made available for query processing before it is ingested into the target source tables. This significantly improves latency issues. The streaming ingest platform can also accommodate uploading data from various sources with different processing and communication capabilities, such as Internet of Things (IOT) devices.
1. A method comprising:
maintaining a source table in a data system;
storing new data from a client device for ingestion into the source table in a storage in a first format;
committing the new data in the storage to make the new data accessible for query processing before the new data is ingested into the source table;
receiving a query;
determining that the query relates to data stored in a source table and committed data for the source table but not ingested in the source table;
converting the committed data from the first format into a common format;
converting the data from the source table from a second format into the common format;
joining the committed data in the common format and the data from the source table in the common format to generate joined data; and
executing a query based on the joined data.
2. The method of claim 1 , further comprising:
retrieving expression properties of the committed data; and
pruning the committed data based on the expression properties and the query.
3. The method of claim 2 , wherein the expression properties include statistics of the committed data.
4. The method of claim 1 , further comprising:
generating a hybrid table based on data stored in the source table and the committed data.
5. The method of claim 1 , wherein the committed data is organized into per-table chunks.
6. The method of claim 5 , wherein the per-table chunks are organized by per-account blobs.
7. The method of claim 1 , wherein the committed data is stored in a different location than the source table.
8. A machine-storage medium embodying instructions that, when executed by a machine, cause the machine to perform operations comprising:
maintaining a source table in a data system;
storing new data from a client device for ingestion into the source table in a storage in a first format;
committing the new data in the storage to make the new data accessible for query processing before the new data is ingested into the source table;
receiving a query;
determining that the query relates to data stored in a source table and committed data for the source table but not ingested in the source table;
converting the committed data from a first format into a common format;
converting the data from the source table from a second format into the common format;
joining the committed data in the common format and the data from the source table in the common format to generate joined data; and
executing a query based on the joined data.
9. The machine-storage medium of claim 8 , further comprising:
retrieving expression properties of the committed data; and
pruning the committed data based on the expression properties and the query.
10. The machine-storage medium of claim 9 , wherein the expression properties include statistics of the committed data.
11. The machine-storage medium of claim 8 , further comprising:
generating a hybrid table based on data stored in the source table and the committed data.
12. The machine-storage medium of claim 8 , wherein the committed data is organized into per-table chunks.
13. The machine-storage medium of claim 12 , wherein the per-table chunks are organized by per-account blobs.
14. The machine-storage medium of claim 8 , wherein the committed data is stored in a different location than the source table.
15. A system comprising:
at least one hardware processor; and
at least one memory storing instructions that cause the at least one hardware processor to perform operations comprising:
maintaining a source table in a data system;
storing new data from a client device for ingestion into the source table in a storage in a first format;
committing the new data in the storage to make the new data accessible for query processing before the new data is ingested into the source table;
receiving a query;
determining that the query relates to data stored in a source table and committed data for the source table but not ingested in the source table;
converting the committed data from a first format into a common format;
converting the data from the source table from a second format into the common format;
joining the committed data in the common format and the data from the source table in the common format to generate joined data; and
executing a query based on the joined data.
16. The system of claim 15 , the operations further comprising:
retrieving expression properties of the committed data; and
pruning the committed data based on the expression properties and the query.
17. The system of claim 16 , wherein the expression properties include statistics of the committed data.
18. The system of claim 15 , the operations further comprising:
generating a hybrid table based on data stored in the source table and the committed data.
19. The system of claim 15 , wherein the committed data is organized into per-table chunks.
20. The system of claim 19 , wherein the per-table chunks are organized by per-account blobs.