Caching metadata for a database table
The subject technology caches, in connection with processing a first query, one or more table-metadata files in at least one of a current version set of one or more table-metadata files and a new version set of one or more table-metadata files, the new version set of one or more table-metadata files comprising table metadata for a new version of a database table. The subject technology receives a subsequent query directed to the database table. The subject technology downloads, in connection with processing the subsequent query, at least one uncached table-metadata file in a scan set of table-metadata files for the subsequent query. The subject technology processes the subsequent query using the at least one uncached table-metadata file.
1. A method comprising:
caching, in connection with processing a first query, one or more table-metadata files in at least one of a current version set of one or more table-metadata files and a new version set of one or more table-metadata files, the new version set of one or more table-metadata files comprising table metadata for a new version of a database table;
receiving a subsequent query directed to the database table;
downloading, in connection with processing the subsequent query, at least one uncached table-metadata file in a scan set of table-metadata files for the subsequent query; and
processing the subsequent query using the at least one uncached table-metadata file.
2. The method of claim 1 , further comprising:
prior to caching the one or more table-metadata files, generating and storing the new version set of one or more table-metadata files in immutable storage.
3. The method of claim 1 , further comprising:
determining that a plurality of table-metadata files are not stored in a cache;
downloading, in parallel, the plurality of table-metadata files from immutable storage; and
storing, in the cache, the plurality of table-metadata files.
4. The method of claim 3 , further comprising:
reading, among the plurality of table-metadata files, a first table-metadata file before a second table-metadata file has been fully downloaded, the plurality of table-metadata files comprising at least the first table-metadata file and the second table-metadata file.
5. The method of claim 4 , wherein reading the first table-metadata file occurs during a process for generating the scan set of table-metadata files.
6. The method of claim 1 , wherein the scan set of table-metadata files indicates at least one table-metadata file for performing the subsequent query.
7. The method of claim 1 , further comprising:
determining that a cache storing the one or more table-metadata files is out of space; and
deleting at least one of the one or more table-metadata files from the cache.
8. The method of claim 7 , wherein the one or more of table-metadata files are cached in a local cache within at least one cluster of an execution platform.
9. The method of claim 1 , further comprising:
determining first version number of a current version set of one or more table-data files;
determining a second version number of a particular version set of the table-metadata files that is smaller or equal to than the first version number; and
generating the scan set of the table-metadata files based on the particular version set of the table-metadata files.
10. The method of claim 9 , wherein the second version number comprises a smaller number than the first version number.
11. A system comprising:
at least one processor; and
a memory device including instructions, which when executed by the at least one processor, cause the at least one processor to perform operations comprising:
caching, in connection with processing a first query, one or more table-metadata files in at least one of a current version set of one or more table-metadata files and a new version set of one or more table-metadata files, the new version set of one or more table-metadata files comprising table metadata for a new version of a database table;
receiving a subsequent query directed to the database table;
downloading, in connection with processing the subsequent query, at least one uncached table-metadata file in a scan set of table-metadata files for the subsequent query; and
processing the subsequent query using the at least one uncached table-metadata file.
12. The system of claim 11 , wherein the operations further comprise:
prior to caching the one or more table-metadata files, generating and storing the new version set of one or more table-metadata files in immutable storage.
13. The system of claim 11 , wherein the operations further comprise:
determining that a plurality of table-metadata files are not stored in a cache;
downloading, in parallel, the plurality of table-metadata files from immutable storage; and
storing, in the cache, the plurality of table-metadata files.
14. The system of claim 13 , wherein the operations further comprise:
reading, among the plurality of table-metadata files, a first table-metadata file before a second table-metadata file has been fully downloaded, the plurality of table-metadata files comprising at least the first table-metadata file and the second table-metadata file.
15. The system of claim 14 , wherein reading the first table-metadata file occurs during a process for generating the scan set of table-metadata files.
16. The system of claim 11 , wherein the scan set of table-metadata files indicates at least one table-metadata file for performing the subsequent query.
17. The system of claim 11 , wherein the operations further comprise:
determining that a cache storing the one or more table-metadata files is out of space; and
deleting at least one of the one or more table-metadata files from the cache.
18. The system of claim 17 , wherein the one or more of table-metadata files are cached in a local cache within at least one cluster of an execution platform.
19. The system of claim 11 , wherein the operations further comprise:
determining first version number of a current version set of one or more table-data files;
determining a second version number of a particular version set of the table-metadata files that is smaller or equal to than the first version number; and
generating the scan set of the table-metadata files based on the particular version set of the table-metadata files.
20. The system of claim 19 , wherein the second version number comprises a smaller number than the first version number.
21. A non-transitory computer-readable medium comprising instructions, which when executed by at least one processor, cause the at least one processor to perform operations comprising:
caching, in connection with processing a first query, one or more table-metadata files in at least one of a current version set of one or more table-metadata files and a new version set of one or more table-metadata files, the new version set of one or more table-metadata files comprising table metadata for a new version of a database table;
receiving a subsequent query directed to the database table;
downloading, in connection with processing the subsequent query, at least one uncached table-metadata file in a scan set of table-metadata files for the subsequent query; and
processing the subsequent query using the at least one uncached table-metadata file.
22. The non-transitory computer-readable medium of claim 21 , wherein the operations further comprise:
prior to caching the one or more table-metadata files, generating and storing the new version set of one or more table-metadata files in immutable storage.
23. The non-transitory computer-readable medium of claim 21 , wherein the operations further comprise:
determining that a plurality of table-metadata files are not stored in a cache;
downloading, in parallel, the plurality of table-metadata files from immutable storage; and
storing, in the cache, the plurality of table-metadata files.
24. The non-transitory computer-readable medium of claim 23 , wherein the operations further comprise:
reading, among the plurality of table-metadata files, a first table-metadata file before a second table-metadata file has been fully downloaded, the plurality of table-metadata files comprising at least the first table-metadata file and the second table-metadata file.
25. The non-transitory computer-readable medium of claim 24 , wherein reading the first table-metadata file occurs during a process for generating the scan set of table-metadata files.
26. The non-transitory computer-readable medium of claim 21 , wherein the scan set of table-metadata files indicates at least one table-metadata file for performing the subsequent query.
27. The non-transitory computer-readable medium of claim 21 , wherein the operations further comprise:
determining that a cache storing the one or more table-metadata files is out of space; and
deleting at least one of the one or more table-metadata files from the cache.
28. The non-transitory computer-readable medium of claim 27 , wherein the one or more of table-metadata files are cached in a local cache within at least one cluster of an execution platform.
29. The non-transitory computer-readable medium of claim 21 , wherein the operations further comprise:
determining first version number of a current version set of one or more table-data files;
determining a second version number of a particular version set of the table-metadata files that is smaller or equal to than the first version number; and
generating the scan set of the table-metadata files based on the particular version set of the table-metadata files.
30. The non-transitory computer-readable medium of claim 29 , wherein the second version number comprises a smaller number than the first version number.