Database metadata in immutable storage
A method for a database system includes storing table data for a database, the table data including information in rows and columns of one or more database tables. The method includes storing metadata on immutable storage, the metadata including information about the table data for the database. In one embodiment, mutable metadata may be periodically consolidated in the background to create new versions of metadata files and which allows for deletions of old metadata files and old data files.
1. A method comprising:
generating and storing a new-version set of one or more table-metadata files, the new-version set of one or more table-metadata files comprising table metadata for a new version of a database table;
caching, in connection with processing a first query, one or more of table-metadata files in at least one of a current-version set of one or more table-metadata files and a new version set of one or more table-metadata files;
receiving a subsequent query directed to the database table;
downloading, in connection with processing the subsequent query, at least one uncached table-metadata file in a scan set of table-metadata files for the subsequent query; and
processing the subsequent query using the at least one uncached table-metadata file.
2. The method of claim 1 , further comprising:
determining that a plurality of table-metadata files are not included in a cache;
downloading, in parallel, the plurality of table-metadata files from immutable storage; and
storing, in the cache, the plurality of table-metadata files.
3. The method of claim 2 , wherein the immutable storage is provided by a cloud storage resource.
4. The method of claim 2 , further comprising:
reading, among the plurality of table-metadata files, a first table-metadata file before a second table-metadata file has been fully downloaded, the plurality of table-metadata files comprising at least the first table-metadata file and the second table-metadata file.
5. The method of claim 4 , wherein reading the first table-metadata file occurs during a process for generating the scan set of table-metadata files.
6. The method of claim 1 , wherein the scan set of table-metadata files indicates at least one table-metadata file for performing the subsequent query.
7. The method of claim 1 , wherein the one or more of table-metadata files are cached in a local cache within at least one cluster of an execution platform.
8. The method of claim 1 , further comprising:
determining first version number of a current version set of one or more table-data files
determining a second version number of a particular version set of the table-metadata files that is smaller or equal to than the first version number; and
generating the scan set of the table-metadata files based on the particular version set of the table-metadata files.
9. The method of claim 8 , wherein the second version number comprises a smaller number than the first version number.
10. The method of claim 9 , wherein the smaller number indicates that at least a portion of one new table-metadata file has yet to be stored in immutable storage.
11. A system comprising:
at least one processor; and
a memory device including instructions, which when executed by the at least one processor, cause the at least one processor to perform operations comprising:
generating and storing a new-version set of one or more table-metadata files, the new-version set of one or more table-metadata files comprising table metadata for a new version of a database table;
caching, in connection with processing a first query, one or more of table-metadata files in at least one of a current-version set of one or more table-metadata files and a new version set of one or more table-metadata files;
receiving a subsequent query directed to the database table;
downloading, in connection with processing the subsequent query, at least one uncached table-metadata file in a scan set of table-metadata files for the subsequent query; and
processing the subsequent query using the at least one uncached table-metadata file.
12. The system of claim 11 , wherein the memory device includes further instructions, which when executed by the at least one processor, cause the at least one processor to perform further operations comprising:
determining that a plurality of table-metadata files are not included in a cache;
downloading, in parallel, the plurality of table-metadata files from immutable storage; and
storing, in the cache, the plurality of table-metadata files.
13. The system of claim 12 , wherein the immutable storage is provided by a cloud storage resource.
14. The system of claim 12 , wherein the memory device includes further instructions, which when executed by the at least one processor, cause the at least one processor to perform further operations comprising:
reading, among the plurality of table-metadata. files, a first table-metadata file before a second table-metadata file has been fully downloaded, the plurality of table-metadata files comprising at least the first table-metadata file and the second table-metadata file.
15. The system of claim 14 , wherein reading the first table-metadata file occurs during a process for generating the scan set of table-metadata files.
16. The system of claim 11 , wherein the scan set of table-metadata files indicates at least one table-metadata file for performing the subsequent query.
17. The system of claim 11 , wherein the one or more of table-metadata files are cached in a local cache within at least one cluster of an execution platform.
18. The system of claim 11 , wherein the memory device includes further instructions, which when executed by the at least one processor, cause the at least one processor to perform further operations comprising:
determining first version number of a current version set of one or more table-data files
determining a second version number of a particular version set of the table-metadata files that is smaller or equal to than the first version number; and
generating the scan set of the table-metadata files based on the particular version set of the table-metadata files.
19. The system of claim 18 , wherein the second version number comprises a smaller number than the first version number.
20. The system of claim 19 , wherein the smaller number indicates that as least a portion of one new table-metadata file has yet to be stored in immutable storage.
21. A non-transitory computer-readable medium comprising instructions, which when executed by at least one processor, cause the at least one processor to perform operations comprising:
generating and storing a new-version set of one or more table-metadata files, the new-version set of one or more table-metadata tiles comprising table metadata for a new version of a database table;
caching, in connection with processing a first query, one or more of table-metadata files in at least one of a current-version set of one or more table-metadata files and a new version set of one or more table-metadata files;
receiving a subsequent query directed to the database table;
downloading, in connection with processing the subsequent query, at least one uncached table-metadata file in a scan set of table-metadata files for the subsequent query; and
processing the subsequent query using the at least one uncached table-metadata file.
22. The non-transitory computer-readable medium of claim 21 , wherein the non-transitory computer-readable medium comprises further instructions, which when executed by the at least one processor, further cause the at least one processor to perform further operations comprising:
determining that a plurality of table-metadata files are not included in a cache;
downloading, in parallel, the plurality of table-metadata files from immutable storage; and
storing, in the cache, the plurality of table-metadata files.
23. The non-transitory computer-readable medium of claim 22 , wherein the immutable storage is provided by a cloud storage resource.
24. The non-transitory computer-readable medium of claim 22 , wherein the non-transitory computer-readable medium comprises further instructions, which when executed by the at least one processor, further cause the at least one processor to perform further operations comprising:
reading, among the plurality of table-metadata files, a first table-metadata file before a second table-metadata file has been fully downloaded, the plurality of table-metadata files comprising at least the first table-metadata file and the second table-metadata file.
25. The non-transitory computer-readable medium of claim 24 , wherein reading the first table-metadata file occurs during a process for generating the scan set of table-metadata files.
26. The non-transitory computer-readable medium of claim 21 , wherein the scan set of table-metadata files indicates at least one table-metadata tile for performing the subsequent query.
27. The non-transitory computer-readable medium of claim 21 , wherein the one or more of table-metadata files are cached in a local cache within at least one cluster of an execution platform.
28. The non-transitory computer-readable medium of claim 21 , wherein the non-transitory computer-readable medium comprises further instructions, which when executed by the at least one processor, further cause the at least one processor to perform further operations comprising:
determining first version number of a current version set of one or more table-data files
determining a second version number of a particular version set of the table-metadata files that is smaller or equal to than the first version number; and
generating the scan set of the table-metadata files based on the particular version set of the table-metadata files.
29. The non-transitory computer-readable medium of claim 28 , wherein the second version number comprises a smaller number than the first version number.
30. The non-transitory computer-readable medium of claim 29 , wherein the smaller number indicates that at least a portion of one new table-metadata file has yet to be stored in immutable storage.