Managing versions of database metadata in storage
A method for a database system includes storing table data for a database, the table data including information in rows and columns of one or more database tables. The method includes storing metadata on immutable storage, the metadata including information about the table data for the database. In one embodiment, mutable metadata may be periodically consolidated in the background to create new versions of metadata files and which allows for deletions of old metadata files and old data files.
1. A method comprising:
storing table metadata for a current version of a database table in a current version set of one or more table-metadata files;
detecting a modification to the current version of the database table, the modification resulting in a new version of the database table, the new version of the database table being stored in a new version set of one or more table-data files; the new version set of one or more table-data files being different than a current version set of one or more table-data files, the table metadata is stored in metadata storage, the new version of the database table is stored in second storage different than the metadata storage; and
in response to detecting the modification to the current version of the database table, generating and storing a new version set of one or more table-metadata files, the new version set of one or more table-metadata files comprising table metadata for the new version of the database table, the new version set of one or more table-metadata files is stored in the metadata storage, the second storage comprises immutable storage, the immutable storage comprises data storage in which data is not updated in place.
2. The method of claim 1 , further comprising:
storing the current version of the database table in the current version set of one or more table-data files, wherein the current version set of one or more table-data files and the new version set of one or more table-data files are also stored in the immutable storage.
3. The method of claim 1 , wherein the metadata storage comprises immutable storage, the immutable storage comprises data storage in which data is not updated in place.
4. The method of claim 1 , wherein the metadata store comprises a key-value store, a main memory storage, or a file system storage.
5. The method of claim 1 , further comprising:
consolidating the new version set of one or more table-metadata files by generating a second new version set of table-metadata files, the second new version set of table-metadata files comprising metadata absent of information from files that have been deleted, the second new version set of table-metadata files being smaller than the new version set of table-metadata files.
6. The method of claim 1 , further comprising:
caching, when processing a first query, at least one of the table-metadata files in at least one of the current version set of one or more table-metadata files and the new version set of one or more table-metadata files;
receiving a subsequent query directed to the database table;
downloading, from the metadata storage, uncached table-metadata files in a scan set of table-metadata files for the subsequent query; and
processing the subsequent query using the table-metadata files that have been downloaded.
7. The method of claim 6 , wherein the scan set comprises an aggregation of delete or added files.
8. The method of claim 6 , wherein the uncached table-metadata files are downloaded in parallel, and a first downloaded table-metadata file is read before a second table-metadata file has been completely downloaded.
9. A system comprising:
a processor; and
a memory device including instructions, which when executed by the processor, cause the processor to perform operations comprising:
storing table metadata for a current version of a database table in a current version current version set of one or more table-metadata files;
detecting a modification to the current version of the database table, the modification resulting in a new version of the database table, the new version of the database table being stored in a new version set of one or more table-data files, the new version set of one or more table-data files being different than a current version set of one or more table-data files, the table metadata is stored in metadata storage, the new version of the database table is stored in second storage different than the metadata storage; and
in response to detecting the modification to the current version of the database table, generating and storing a new version set of one or more table-metadata files, the new version set of one or more table-metadata files comprising table metadata for the new version of the database table, the new version set of one or more table-metadata files is stored in the metadata storage, the second storage comprises immutable storage, the immutable storage comprises data storage in which data is not updated in place.
10. The system of claim 9 , wherein the memory device includes further instructions, which when executed by the processor, cause the processor to perform further operations comprising:
storing the current version of the database table in the current version set of one or more table-data files, wherein the current version set of one or more table-data files and the new version set of one or more table-data files are also stored in the immutable storage.
11. The system of claim 9 , wherein the metadata storage comprises immutable storage, the immutable storage comprises data storage in which data is not updated in place.
12. The system of claim 9 , wherein the metadata store comprises a key-value store, a main memory storage, or a file system storage.
13. The system of claim 9 , wherein the memory device includes further instructions, which when executed by the processor, cause the processor to perform further operations comprising:
consolidating the new version set of one or more table-metadata files by generating a second new version set of table-metadata files, the second new version set of table-metadata files comprising metadata absent of information from files that have been deleted, the second new version set of table-metadata files being smaller than the new version set of table-metadata files.
14. The system of claim 9 , wherein the memory device includes further instructions, which when executed by the processor, cause the processor to perform further operations comprising:
caching, when processing a first query, at least one of the table-metadata files in at least one of the current version set of one or more table-metadata files and the new version set of one or more table-metadata files;
receiving a subsequent query directed to the database table;
downloading, from the metadata storage, uncached table-metadata files in a scan set of table-metadata files for the subsequent query; and
processing the subsequent query using the table-metadata files that have been downloaded.
15. The system of claim 14 , wherein the scan set comprises an aggregation of deleted or added files.
16. The system of claim 14 , wherein the uncached table-metadata files are downloaded in parallel, and a first downloaded table-metadata file is read before a second table-metadata file has been completely downloaded.
17. A non-transitory computer-readable medium comprising instructions, which when executed by a processor, cause the processor to perform operations comprising:
storing table metadata for a current version of a database table in a current version set of one or more table-metadata files;
detecting a modification to the current version of the database table, the modification resulting in a new version of the database table, the new version of the database table being stored in a new version set of one or more table-data files, the new version set of one or more table-data files being different than a current version set of one or more table-data files, the table metadata is stored in metadata storage, the new version of the database table is stored in second storage different than the metadata storage; and
in response to detecting the modification to the current version of the database table, generating and storing a new version set of one or more table-metadata files, the new version set of one or more table-metadata files comprising table metadata for the new version of the database table, the new version set of one or more table-metadata files is stored in the metadata storage the second storage comprises immutable storage, the immutable storage comprises data storage in which data is not updated in place.
18. The non-transitory computer-readable medium of claim 17 , wherein the non-transitory computer-readable medium comprises further instructions, which when executed by the processor, further cause the processor to perform further operations comprising:
storing the current version of the database table in the current version set of one or more table-data files, wherein the current version set of one or more table-data files and the new version set of one or more table-data files are also stored in the immutable storage.
19. The non-transitory computer-readable medium of claim 17 , wherein the metadata storage comprises immutable storage, the immutable storage comprises data storage in which data is not updated in place.
20. The non-transitory computer-readable medium of claim 17 , wherein the metadata store comprises a key-value store, a main memory storage, or a file system storage.
21. The non-transitory computer-readable medium of claim 17 , wherein the non-transitory computer-readable medium comprises further instructions, which when executed by the processor, further cause the processor to perform further operations comprising:
consolidating the new version set of one or more table-metadata files by generating a second new version set of table-metadata files, the second new version set of table-metadata files comprising metadata absent of information from files that have been deleted, the second new version set of table-metadata files being smaller than the new version set of table-metadata files.
22. The non-transitory computer-readable medium of claim 17 , wherein the non-transitory computer-readable medium comprises further instructions, which when executed by the processor, further cause the processor to perform further operations comprising:
caching, when processing a first query, at least one of the table-metadata files in at least one of the current version set of one or more table-metadata files and the new version set of one or more table-metadata files;
receiving a subsequent query directed to the database table;
downloading, from the metadata storage, uncached table-metadata files in a scan set of table-metadata files for the subsequent query; and
processing the subsequent query using the table-metadata files that have been downloaded.
23. The non-transitory computer-readable medium of claim 22 , wherein the scan set comprises an aggregation of deleted or added files.
24. The non-transitory computer-readable medium of claim 22 , wherein the uncached table-metadata files are downloaded in parallel, and a first downloaded table-metadata file is read before a second table-metadata file has been completely downloaded.