Virtual database system
View Patent ↗Information from multiple databases is retrieved and stored on a database storage system. Multiple point-in-time copies are obtained for each database. A point-in-time copy retrieves data changed in the database since the retrieval of a previous point-in-time copy. A virtual database (VDB) is created by creating a set of files in the data storage system. Each file in the set of files created for a VDB is linked to the database blocks on the database storage system associated with a point-in-time copy of the source database. The set of files associated with the VDB are mounted on a database server allowing the database server to read from and write to the set of files. Workflows based on VDBs allow various usage scenarios based on databases to be implemented efficiently, for example, testing and development, backup and recovery, and data warehouse building.
1. A method for creating a virtual database system, the method comprising:
receiving, by a storage system, a plurality of point-in-time copies of a source database, the source database comprising a plurality of database blocks, each point-in-time copy comprising one or more database blocks, wherein each database block includes metadata associated with the database block;
storing, on the storage system, database blocks for a plurality of point-in-time copies of the source database;
receiving, by the storage system, transaction logs from the source database, the transaction logs received independent of the point-in-time copies, wherein the transaction logs represent changes to the source database since a previous point-in-time copy was received;
receiving a request to create a virtual database, the request specifying a point in time;
identifying a point-in-time copy stored in the storage system, the point-in-time copy obtained before the specified point in time;
modifying database blocks of the identified point-in-time copy by applying one or more transaction logs received from the source database after receiving the identified point-in-time copy;
creating a virtual database comprising a plurality of database blocks stored in the storage system, the plurality of database blocks comprising at least a modified database block of the identified point-in-time copy and a second database block of another point-in-time copy; and
providing access to the virtual database from a server allowing the server to read from and write to the database blocks of the virtual database.
2. The method of claim 1 , wherein the transaction logs are retrieved from the source database according to a first schedule and the point-in-time copies are retrieved according to a second schedule distinct from the first schedule.
3. The method of claim 1 , wherein the transaction logs are retrieved at a frequency higher than the frequency at which the point-in-time copies are retrieved.
4. The method of claim 1 , wherein receiving point-in-time copies comprises receiving data streams corresponding to the point-in-time copies, wherein each data stream comprises data from database blocks associated with the source database, the method further comprising, analyzing the data streams received to identify database blocks.
5. The method of claim 4 , further comprising:
analyzing the metadata of database blocks to determine length of the database blocks for processing the data stream.
6. The method of claim 4 , further comprising:
analyzing the metadata of database blocks to determine whether the database block is stored.
7. The method of claim 6 , further comprising:
determining not to store a database block responsive to determining that the database block is one of a temporary database block or an empty database block.
8. The method of claim 1 , wherein metadata associated with the database block comprises information indicating a time of update of data of the database block.
9. The method of claim 1 , wherein metadata associated with the database block comprises information related to the data stored in the database block.
10. The method of claim 1 , wherein metadata associated with the database block comprises information related to one or more objects of a database that the database block is part of.
11. The method of claim 1 , wherein the metadata associated with a database block comprises a size of the database block.
12. The method of claim 1 , wherein the metadata associated with a database block comprises a log sequence number based on an order in which database blocks are updated in the source database.
13. The method of claim 1 , further comprising:
transforming, by the storage system, data of a received database block to a format distinct from a format of the received database block and storing the transformed data instead of the received block.
14. The method of claim 13 , wherein transforming the data of a received database block comprises compressing data stored in the received database block.
15. The method of claim 13 , wherein transforming the data of a received database block comprises masking data stored in the received database block.
16. A non-transitory computer-readable storage medium storing instructions for:
receiving, by a storage system, a plurality of point-in-time copies of a source database, the source database comprising a plurality of database blocks, each point-in-time copy comprising one or more database blocks, wherein each database block includes metadata associated with the database block;
storing, on the storage system, database blocks for a plurality of point-in-time copies of the source database;
receiving, by the storage system, transaction logs from the source database, the transaction logs received independent of the point-in-time copies, wherein the transaction logs represent changes to the source database since a previous point-in-time copy was received;
receiving a request to create a virtual database, the request specifying a point in time;
identifying a point-in-time copy stored in the storage system, the point-in-time copy obtained before the specified point in time;
modifying database blocks of the identified point-in-time copy by applying one or more transaction logs received from the source database after receiving the identified point-in-time copy;
creating a virtual database comprising a plurality of database blocks stored in the storage system, the plurality of database blocks comprising at least a modified database block of the identified point-in-time copy and a second database block of another point-in-time copy; and
providing access to the virtual database from a server allowing the server to read from and write to the database blocks of the virtual database.
17. The non-transitory computer-readable storage medium of claim 16 , wherein the transaction logs are retrieved from the source database according to a first schedule and the point-in-time copies are retrieved according to a second schedule distinct from the first schedule.
18. The non-transitory computer-readable storage medium of claim 16 , wherein the transaction logs are retrieved at a frequency higher than the frequency at which the point-in-time copies are retrieved.
19. The non-transitory computer-readable storage medium of claim 16 , wherein receiving point-in-time copies comprises receiving data streams corresponding to the point-in-time copies, wherein each data stream comprises data from database blocks associated with the source database, wherein the stored instructions are further for:
analyzing the data streams received to identify database blocks.
20. The non-transitory computer-readable storage medium of claim 16 , wherein the stored instructions are further for:
analyzing the metadata of database blocks to determine whether the database block is stored and determining not to store a database block responsive to determining that the database block is one of a temporary database block or an empty database block.
21. The non-transitory computer-readable storage medium of claim 16 , wherein metadata associated with the database block comprises one or more of:
information indicating a time of update of data of the database block;
information related to the data stored in the database block;
information related to one or more objects of a database that the database block is part of;
a size of the database block; or
a log sequence number based on an order in which database blocks are updated in the source database.
22. The non-transitory computer-readable storage medium of claim 16 , wherein the stored instructions are further for:
transforming, by the storage system, data of a received database block to a format distinct from a format of the received database block and storing the transformed data instead of the received block.
23. The non-transitory computer-readable storage medium of claim 22 , wherein transforming the data of a received database block comprises one or more of: compressing data stored in the received database block or masking data stored in the received database block.
24. A computer system comprising:
a computer processor; and
non-transitory computer-readable storage medium storing instructions for executing by the computer processor, the instructions for:
receiving, by a storage system, a plurality of point-in-time copies of a source database, the source database comprising a plurality of database blocks, each point-in-time copy comprising one or more database blocks, wherein each database block includes metadata associated with the database block;
storing, on the storage system, database blocks for a plurality of point-in-time copies of the source database;
receiving, by the storage system, transaction logs from the source database, the transaction logs received independent of the point-in-time copies, wherein the transaction logs represent changes to the source database since a previous point-in-time copy was received;
receiving a request to create a virtual database, the request specifying a point in time;
identifying a point-in-time copy stored in the storage system, the point-in-time copy obtained before the specified point in time;
modifying database blocks of the identified point-in-time copy by applying one or more transaction logs received from the source database after receiving the identified point-in-time copy;
creating a virtual database comprising a plurality of database blocks stored in the storage system, the plurality of database blocks comprising at least a modified database block of the identified point-in-time copy and a second database block of another point-in-time copy; and
providing access to the virtual database from a server allowing the server to read from and write to the database blocks of the virtual database.
25. The computer system of claim 24 , wherein the transaction logs are retrieved from the source database according to a first schedule and the point-in-time copies are retrieved according to a second schedule distinct from the first schedule.
26. The computer system of claim 24 , wherein the transaction logs are retrieved at a frequency higher than the frequency at which the point-in-time copies are retrieved.
27. The computer system of claim 24 , wherein receiving point-in-time copies comprises receiving data streams corresponding to the point-in-time copies, wherein each data stream comprises data from database blocks associated with the source database, wherein the instructions stored in the non-transitory computer-readable storage medium are further for:
analyzing the data streams received to identify database blocks.
28. The computer system of claim 24 , wherein the instructions stored in the non-transitory computer-readable storage medium are further for:
analyzing the metadata of database blocks to determine whether the database block is stored and determining not to store a database block responsive to determining that the database block is one of a temporary database block or an empty database block.
29. The computer system of claim 24 , wherein metadata associated with the database block comprises one or more of:
information indicating a time of update of data of the database block;
information related to the data stored in the database block;
information related to one or more objects of a database that the database block is part of;
a size of the database block; or
a log sequence number based on an order in which database blocks are updated in the source database.
30. The computer system of claim 24 , wherein the instructions stored in the non-transitory computer-readable storage medium are further for:
transforming, by the storage system, data of a received database block to a format distinct from a format of the received database block and storing the transformed data instead of the received block.
31. The computer system of claim 30 , wherein transforming the data of a received database block comprises one or more of: compressing data stored in the received database block or masking data stored in the received database block.