Virtual partitions in virtual databases
Information from source databases is retrieved and stored on a database storage system. Multiple point-in-time copies are obtained for each database. A source database may contain partitions comprising sets of database tables. A partition is imported into a virtual database created. The database storage system validates the partition before importing, by checking whether the partition includes database tables that refer to database tables outside the partition. A partition imported in a virtual database may be rewinded to an older state or refreshed to a more recent state. Multiple partitions may be included in a virtual database and the state of each partition modified independent of the state of the remaining virtual database including the other partitions. Multiple versions of the same partition corresponding to different points in time are incorporated in the same virtual database, thereby allowing a query to process different versions of the same database table.
1. A method for creating a virtual database system, the method comprising:
storing on a storage system, database blocks for a plurality of snapshots of a source database, wherein the source database stores a partition comprising a set of database tables, wherein each snapshot represents a state of the source database at a point in time, wherein at least some of the stored database blocks of a snapshot are shared with one or more other snapshots of the source database;
creating a virtual database represented by a set of files, the virtual database sharing one or more database blocks with one or more other virtual databases;
importing a plurality of versions of the partition into the virtual database comprising:
importing a first version of the partition into the virtual database comprising:
updating the metadata of the virtual database to describe database objects of the first version of the partition; and
modifying the set of files of the virtual database to point at database blocks of the first version of the partition stored in the storage system;
importing a second version of the partition into the virtual database comprising:
updating the metadata of the virtual database to describe database objects of the second version of the partition; and
modifying the set of files of the virtual database to point at database blocks of the second version of the partition stored in the storage system, wherein the database blocks of the second version of the partition share one or more database blocks with the first version of the partition; and
processing data of the plurality of versions of the partitions in response to database queries received by the virtual database.
2. The method of claim 1 , wherein processing data of the plurality of versions of the partitions in response to database queries received by the virtual database comprises:
receiving a query identifying a particular table, the query processing data of the particular table from the first version of the partition and data of the particular table from the second version of the partition; and
executing the query to generate a result set based on the data of the particular table from the first version of the partition and the data of the particular table from the second version of the partition.
3. The method of claim 1 , wherein executing the query comprises:
receiving a request to load in memory, one or more database blocks of the table from the first version of the partition; and
loading in memory, one or more database blocks of the table from the first version of the partition;
receiving a request to load in memory one or more database blocks of the table from the second version of the partition; and
determining whether the one or more database blocks of the table from the second version of the partition match the one or more database blocks of the table from the first version of the partition previously loaded in memory; and
responsive to determining that the one or more database blocks of the table from the second version of the partition match the previously loaded one or more database blocks of the table from the first version of the partition, reusing the previously loaded one or more database blocks of the table from the first version of the partition as database blocks of the table from the second version of the partition.
4. The method of claim 1 , further comprising:
validating the partition by checking if the database tables in the partition refer to database tables outside the partition; and
responsive to determining that a database table in the partition refers to a database table outside the partition, returning error indicating failure to perform import operation.
5. The method of claim 4 , wherein validating a partition comprises:
creating a staging virtual database based on the point in time associated with the partition; and
checking if the database tables in the partition refer to database tables outside the partition using the staging virtual database.
6. The method of claim 1 , further comprising:
performing import of a third version of the partition into the virtual database comprising:
updating the metadata of the virtual database to describe database objects of the third version of the partition; and
modifying the set of files of the virtual database to point at database blocks of the third version of the partition stored in the storage system, wherein the database blocks of the third version of the partition share one or more database blocks with the first version of the partition and the second version of the partition.
7. The method of claim 1 , wherein the partition is a first partition and the source database comprises a second partition, the method further comprising:
performing import of a version of the second partition into the virtual database comprising:
updating the metadata of the virtual database to describe database objects of the version of the second partition; and
modifying the set of files of the virtual database to point at database blocks of the version of the second partition stored in the storage system, wherein the database blocks of the version of the second partition share one or more database blocks with other versions of the second partition.
8. A non-transitory computer-readable storage medium storing instructions for:
storing on a storage system, database blocks for a plurality of snapshots of a source database, wherein the source database stores a partition comprising a set of database tables, wherein each snapshot represents a state of the source database at a point in time, wherein at least some of the stored database blocks of a snapshot are shared with one or more other snapshots of the source database;
creating a virtual database represented by a set of files, the virtual database sharing one or more database blocks with one or more other virtual databases;
importing a plurality of versions of the partition into the virtual database comprising:
importing a first version of the partition into the virtual database comprising:
updating the metadata of the virtual database to describe database objects of the first version of the partition; and
modifying the set of files of the virtual database to point at database blocks of the first version of the partition stored in the storage system;
importing a second version of the partition into the virtual database comprising:
updating the metadata of the virtual database to describe database objects of the second version of the partition; and
modifying the set of files of the virtual database to point at database blocks of the second version of the partition stored in the storage system, wherein the database blocks of the second version of the partition share one or more database blocks with the first version of the partition; and
processing data of the plurality of versions of the partitions in response to database queries received by the virtual database.
9. The non-transitory computer-readable storage medium of claim 8 , wherein instructions for processing data of the plurality of versions of the partitions in response to database queries received by the virtual database comprise instructions for:
receiving a query identifying a particular table, the query processing data of the particular table from the first version of the partition and data of the particular table from the second version of the partition; and
executing the query to generate a result set based on the data of the particular table from the first version of the partition and the data of the particular table from the second version of the partition.
10. The non-transitory computer-readable storage medium of claim 8 , wherein instructions for executing the query comprise instructions for:
receiving a request to load in memory, one or more database blocks of the table from the first version of the partition; and
loading in memory, one or more database blocks of the table from the first version of the partition;
receiving a request to load in memory one or more database blocks of the table from the second version of the partition; and
determining whether the one or more database blocks of the table from the second version of the partition match the one or more database blocks of the table from the first version of the partition previously loaded in memory; and
responsive to determining that the one or more database blocks of the table from the second version of the partition match the previously loaded one or more database blocks of the table from the first version of the partition, reusing the previously loaded one or more database blocks of the table from the first version of the partition as database blocks of the table from the second version of the partition.
11. The non-transitory computer-readable storage medium of claim 8 , wherein the stored instructions further comprise instruction for:
validating the partition by checking if the database tables in the partition refer to database tables outside the partition; and
responsive to determining that a database table in the partition refers to a database table outside the partition, returning error indicating failure to perform import operation.
12. The non-transitory computer-readable storage medium of claim 11 , wherein instructions for validating a partition comprise instructions for:
creating a staging virtual database based on the point in time associated with the partition; and
checking if the database tables in the partition refer to database tables outside the partition using the staging virtual database.
13. The non-transitory computer-readable storage medium of claim 8 , wherein the stored instructions further comprise instruction for:
performing import of a third version of the partition into the virtual database comprising:
updating the metadata of the virtual database to describe database objects of the third version of the partition; and
modifying the set of files of the virtual database to point at database blocks of the third version of the partition stored in the storage system, wherein the database blocks of the third version of the partition share one or more database blocks with the first version of the partition and the second version of the partition.
14. The non-transitory computer-readable storage medium of claim 8 , wherein the partition is a first partition and the source database comprises a second partition, wherein the stored instructions further comprise instruction for:
performing import of a version of the second partition into the virtual database comprising:
updating the metadata of the virtual database to describe database objects of the version of the second partition; and
modifying the set of files of the virtual database to point at database blocks of the version of the second partition stored in the storage system, wherein the database blocks of the version of the second partition share one or more database blocks with other versions of the second partition.
15. A computer system comprising:
one or more processors; and
a non-transitory computer-readable storage medium storing instructions for:
storing on a storage system, database blocks for a plurality of snapshots of a source database, wherein the source database stores a partition comprising a set of database tables, wherein each snapshot represents a state of the source database at a point in time, wherein at least some of the stored database blocks of a snapshot are shared with one or more other snapshots of the source database;
creating a virtual database represented by a set of files, the virtual database sharing one or more database blocks with one or more other virtual databases;
importing a plurality of versions of the partition into the virtual database comprising:
importing a first version of the partition into the virtual database comprising:
updating the metadata of the virtual database to describe database objects of the first version of the partition; and
modifying the set of files of the virtual database to point at database blocks of the first version of the partition stored in the storage system;
importing a second version of the partition into the virtual database comprising:
updating the metadata of the virtual database to describe database objects of the second version of the partition; and
modifying the set of files of the virtual database to point at database blocks of the second version of the partition stored in the storage system, wherein the database blocks of the second version of the partition share one or more database blocks with the first version of the partition; and
processing data of the plurality of versions of the partitions in response to database queries received by the virtual database.
16. The computer system of claim 15 , wherein instructions for processing data of the plurality of versions of the partitions in response to database queries received by the virtual database comprise instructions for:
receiving a query identifying a particular table, the query processing data of the particular table from the first version of the partition and data of the particular table from the second version of the partition; and
executing the query to generate a result set based on the data of the particular table from the first version of the partition and the data of the particular table from the second version of the partition.
17. The computer system of claim 15 , wherein instructions for executing the query comprise instructions for:
receiving a request to load in memory, one or more database blocks of the table from the first version of the partition; and
loading in memory, one or more database blocks of the table from the first version of the partition;
receiving a request to load in memory one or more database blocks of the table from the second version of the partition; and
determining whether the one or more database blocks of the table from the second version of the partition match the one or more database blocks of the table from the first version of the partition previously loaded in memory; and
responsive to determining that the one or more database blocks of the table from the second version of the partition match the previously loaded one or more database blocks of the table from the first version of the partition, reusing the previously loaded one or more database blocks of the table from the first version of the partition as database blocks of the table from the second version of the partition.
18. The computer system of claim 15 , wherein the stored instructions further comprise instruction for:
validating the partition by checking if the database tables in the partition refer to database tables outside the partition; and
responsive to determining that a database table in the partition refers to a database table outside the partition, returning error indicating failure to perform import operation.
19. The computer system of claim 18 , wherein instructions for validating a partition comprise instructions for:
creating a staging virtual database based on the point in time associated with the partition; and
checking if the database tables in the partition refer to database tables outside the partition using the staging virtual database.
20. The computer system of claim 15 , wherein the stored instructions further comprise instruction for:
performing import of a third version of the partition into the virtual database comprising:
updating the metadata of the virtual database to describe database objects of the third version of the partition; and
modifying the set of files of the virtual database to point at database blocks of the third version of the partition stored in the storage system, wherein the database blocks of the third version of the partition share one or more database blocks with the first version of the partition and the second version of the partition.
21. The computer system of claim 15 , wherein the partition is a first partition and the source database comprises a second partition, wherein the stored instructions further comprise instruction for:
performing import of a version of the second partition into the virtual database comprising:
updating the metadata of the virtual database to describe database objects of the version of the second partition; and
modifying the set of files of the virtual database to point at database blocks of the version of the second partition stored in the storage system, wherein the database blocks of the version of the second partition share one or more database blocks with other versions of the second partition.