Managed tables for data lakes
Aspects of the disclosure are directed to merging data lake openness with scalable metadata for managed tables in a cloud database platform, allowing for atomicity, consistency, isolation, and durability (ACID) transactions, performant data manipulation language (DML), higher throughput stream ingestion, data consistency, schema evolution, time travel, clustering, fine-grained security, and/or automatic storage optimization. Table data is stored in various open-source file formats in cloud storage while physical metadata of the table data is stored in a scalable metadata storage system.
1. A method for processing queries, comprising:
receiving, by one or more processors, a request from a query engine to write one or more tuples;
writing, by the one or more processors, the one or more tuples to a write-optimized storage in a row-oriented format in a distributed file system that supports file appends;
converting, by the one or more processors, the one or more tuples to one or more data files in a columnar-oriented format compatible with the query engine;
storing, by the one or more processors, the one or more data files in a read-optimized cloud storage in the columnar-oriented format compatible with the query engine; and
committing, by the one or more processors, the write as an addition to a table transaction log stored in the distributed file system by writing a row-level addition to the table transaction log.
2. The method of claim 1 , further comprising performing, by the one or more processors, one or more maintenance tasks in the distributed file system.
3. The method of claim 2 , wherein the one or more maintenance tasks comprise garbage collection of one or more tuples stored in the write-optimized storage.
4. The method of claim 1 , further comprising:
converting, by one or more processors, the table transaction log to the columnar-oriented format compatible with the query engine; and
storing, by one or more processors, the table transaction log in the read-optimized cloud storage compatible with the query engine.
5. The method of claim 1 , wherein the query engine is a data lake query engine.
6. The method of claim 1 , wherein the committing of the write to the table transaction log occurs exactly once.
7. The method of claim 1 , further comprising:
receiving, by the one or more processors, a request from a query engine to read the one or more data files; and
reading, by the one or more processors, the one or more data files through an API that provides read after write semantics.
8. A system comprising:
one or more processors; and
one or more storage devices coupled to the one or more processors and storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations for processing queries, the operations comprising:
receiving a request from a query engine to write one or more tuples;
writing tuples of the one or more tuples to a write-optimized storage in a row-oriented format in a distributed file system that supports file appends;
converting the one or more tuples to one or more data files in a columnar-oriented format compatible with the query engine;
storing the one or more data files in a read-optimized cloud storage in the columnar-oriented format compatible with the query engine; and
committing the write as an addition to a table transaction log stored in the distributed file system by writing a row-level addition to the table transaction log.
9. The system of claim 8 , wherein the operations further comprise performing one or more maintenance tasks in the distributed file system.
10. The system of claim 9 , wherein the one or more maintenance tasks comprise garbage collection of one or more tuples stored in the write-optimized storage.
11. The system of claim 8 , wherein the operations further comprise:
converting the table transaction log to the columnar-oriented format compatible with the query engine; and
storing the table transaction log in the read-optimized cloud storage compatible with the query engine.
12. The system of claim 8 , wherein the query engine is a data lake query engine.
13. The system of claim 8 , wherein the committing of the write to the table transaction log occurs exactly once.
14. The system of claim 8 , wherein the operations further comprise:
receiving a request from a query engine to read one or more additional data files; and
reading the one or more additional data files through an API that provides read after write semantics.
15. A non-transitory computer readable medium for storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations for processing queries, the operations comprising:
receiving a request from a query engine to write one or more tuples;
writing the one or more tuples to a write-optimized storage in a row-oriented format in a distributed file system that supports file appends;
converting the one or more tuples to one or more data files in a columnar-oriented format compatible with the query engine;
storing the one or more data files in a read-optimized cloud storage in the columnar-oriented format compatible with the query engine; and
committing the write as an addition to a table transaction log stored in the distributed file system by writing a row-level addition to the table transaction log.
16. The non-transitory computer readable medium of claim 15 , wherein the operations further comprise performing one or more maintenance tasks in the distributed file system.
17. The non-transitory computer readable medium of claim 16 , wherein the one or more maintenance tasks comprise garbage collection of one or more tuples stored in the write-optimized storage.
18. The non-transitory computer readable medium of claim 15 , wherein the operations further comprise:
converting the table transaction log to the columnar-oriented format compatible with the query engine; and
storing the table transaction log in the read-optimized cloud storage compatible with the query engine.
19. The non-transitory computer readable medium of claim 15 , wherein the committing of the write to the table transaction log occurs exactly once.
20. The non-transitory computer readable medium of claim 15 , wherein the operations further comprise:
receiving a request from a query engine to read one or more additional data files; and
reading the one or more additional data files through an API that provides read after write semantics.