Virtual database system
Information from multiple databases is retrieved and stored on a database storage system. Multiple point-in-time copies are obtained for each database. A point-in-time copy retrieves data changed in the database since the retrieval of a previous point-in-time copy. A virtual database (VDB) is created by creating a set of files in the data storage system. Each file in the set of files created for a VDB is linked to the database blocks on the database storage system associated with a point-in-time copy of the source database. The set of files associated with the VDB are mounted on a database server allowing the database server to read from and write to the set of files. Workflows based on VDBs allow various usage scenarios based on databases to be implemented efficiently, for example, testing and development, backup and recovery, and data warehouse building.
1. A method for creating virtual databases comprising database blocks, wherein each database block includes metadata associated with the database block, the method comprising:
receiving, from a database system, a first data stream comprising a first plurality of database blocks corresponding to a database at a first point in time, wherein each database block of the first data stream includes metadata associated with the database block;
storing on a storage system, the first plurality of database blocks received as part of the first data stream;
creating a first virtual database corresponding to the database at the first point in time, the first virtual database comprising the first plurality of database blocks stored in the storage system;
receiving, from the database system, a second data stream comprising a second plurality of database blocks corresponding to the database at a second point in time that is after the first point in time, wherein each database block of the second data stream includes metadata associated with the database block;
in response to determining that a database block of the second plurality of database blocks has not been updated since the first point in time, not storing the database block in the storage system;
storing a remaining portion of the second plurality of database blocks received as part of the second data stream;
creating a second virtual database corresponding to the database at the second point in time, the second virtual database comprising the remaining portion of the second plurality of database blocks and at least one database block of the first plurality of database blocks;
providing access to the virtual database from a server allowing the server to read from and write to the database blocks of the virtual database.
2. The method of claim 1 , further comprising receiving a plurality of transaction logs corresponding to changes to database blocks of the first plurality of database blocks on the database system between the first point in time and the second point in time.
3. The method of claim 2 , further comprising:
receiving a request for creation of a third virtual database corresponding to a third point in time between the first and second points in time;
identifying a portion of the plurality of transaction logs corresponding to a time period between the first point in time and the third point in time; and
updating one or more stored database blocks of the first plurality of database blocks using the portion of the plurality of transaction logs.
4. The method of claim 3 , wherein updating the one or more stored database blocks comprises:
generating a copy of a stored database block of the one or more stored database blocks; and
applying at least one transaction log of the portion of the plurality of transaction logs to the copy of the stored database block.
5. The method of claim 1 , wherein each of the first and second plurality of database blocks comprises corresponding metadata, the metadata corresponding to a database block comprising information indicating a time at which the database block was last updated.
6. The method of claim 1 , further comprising, responsive to determining that a database block of the second plurality of database blocks is one of a temporary database block or an empty database block, not storing the database block in the storage system.
7. The method of claim 1 , further comprising, transforming, by the storage system, data of a received database block of the first or second data streams to a format distinct from a format of the received database block and storing the transformed data instead of the received block.
8. The method of claim 7 , wherein transforming the data of the received database block comprises compressing and encrypting the data of the received database block.
9. The method of claim 1 , wherein providing read and write access to the first virtual database and the second virtual database comprises:
providing access to the first virtual database from a first database server allowing the first database server to read from and write to the database blocks of the first virtual database; and
providing access to the second virtual database from a second database server allowing the second database server to read from and write to the database blocks of the second virtual database.
10. A non-transitory computer-readable storage medium storing instructions for:
receiving, from a database system, a first data stream comprising a first plurality of database blocks corresponding to a database at a first point in time, wherein each database block of the first data stream includes metadata associated with the database block;
storing on a storage system, the first plurality of database blocks received as part of the first data stream;
creating a first virtual database corresponding to the database at the first point in time, the first virtual database comprising the first plurality of database blocks stored in the storage system;
receiving, from the database system, a second data stream comprising a second plurality of database blocks corresponding to the database at a second point in time that is after the first point in time, wherein each database block of the second data stream includes metadata associated with the database block;
in response to determining that a database block of the second plurality of database blocks has not been updated since the first point in time, not storing the database block in the storage system;
storing a remaining portion of the second plurality of database blocks received as part of the second data stream;
creating a second virtual database corresponding to the database at the second point in time, the second virtual database comprising the remaining portion of the second plurality of database blocks and at least one database block of the first plurality of database blocks;
providing access to the virtual database from a server allowing the server to read from and write to the database blocks of the virtual database.
11. The non-transitory computer-readable storage medium of claim 10 , further comprising instructions for receiving a plurality of transaction logs corresponding to changes to database blocks of the first plurality of database blocks on the database system between the first point in time and the second point in time.
12. The non-transitory computer-readable storage medium of claim 11 , further comprising instructions for:
receiving a request for creation of a third virtual database corresponding to a third point in time between the first and second points in time;
identifying a portion of the plurality of transaction logs corresponding to a time period between the first point in time and the third point in time; and
updating one or more stored database blocks of the first plurality of database blocks using the portion of the plurality of transaction logs.
13. The non-transitory computer-readable storage medium of claim 12 , wherein updating the one or more stored database blocks comprises:
generating a copy of a stored database block of the one or more stored database blocks; and
applying at least one transaction log of the portion of the plurality of transaction logs to the copy of the stored database block.
14. The non-transitory computer-readable storage medium of claim 10 , wherein each of the first and second plurality of database blocks comprises corresponding metadata, the metadata corresponding to a database block comprising information indicating a time at which the database block was last updated.
15. The non-transitory computer-readable storage medium of claim 10 , further comprising instructions for, responsive to determining that a database block of the second plurality of database blocks is one of a temporary database block or an empty database block, not storing the database block in the storage system.
16. The non-transitory computer-readable storage medium of claim 10 , further comprising instructions for, transforming, by the storage system, data of a received database block of the first or second data streams to a format distinct from a format of the received database block and storing the transformed data instead of the received block.
17. The non-transitory computer-readable storage medium of claim 16 , wherein transforming the data of the received database block comprises compressing and encrypting the data of the received database block.
18. The non-transitory computer-readable storage medium of claim 10 , wherein providing read and write access to the first virtual database and the second virtual database comprises:
providing access to the first virtual database from a first database server allowing the first database server to read from and write to the database blocks of the first virtual database; and
providing access to the second virtual database from a second database server allowing the second database server to read from and write to the database blocks of the second virtual database.
19. A method for creating a virtual database system, the method comprising:
generating, at a database system, a first data stream comprising a first plurality of database blocks corresponding to a database at a first point in time, wherein each database block of the second data stream includes metadata associated with the database block;
transmitting the first data stream from the database system to a storage system, wherein the storage system creates a first virtual database corresponding to the database at the first point in time, the first virtual database comprising the first plurality of database blocks stored in the storage system;
generating, at the database system, a second data stream comprising a second plurality of database blocks corresponding to the database at a second point in time that is after the first point in time, wherein each database block of the second data stream includes metadata associated with the database block;
in response to determining that a database block of the second plurality of database blocks has not been updated since the first point in time, removing the database block from the second data stream;
transmitting the second data stream having a remaining portion of the second plurality of database blocks to the storage system, wherein the storage system creates a second virtual database corresponding to the database at the second point in time, the second virtual database comprising the remaining portion of the second plurality of database blocks and at least one database block of the first plurality of database blocks and provides access to the virtual database from a server allowing the server to read from and write to the database blocks of the virtual database.
20. The method of claim 19 , further comprising transmitting to the storage system a plurality of transaction logs corresponding to changes to database blocks of the first plurality of database blocks on the database system between the first point in time and the second point in time.
21. The method of claim 19 , wherein each of the first and second plurality of database blocks comprises corresponding metadata, the metadata corresponding to a database block comprising information indicating a time at which the database block was last updated.
22. The method of claim 19 , further comprising, responsive to determining that a database block of the second plurality of database blocks is one of a temporary database block or an empty database block, removing the database block from the second data stream.