Global de-duplication of virtual disks in a storage platform
In order to avoid writing duplicates of blocks of data into a storage platform, any virtual disk within the storage platform may have a de-duplication feature enabled. Or, all virtual disks have this feature enabled. For virtual disks with de-duplication enabled, a unique message digest is calculated for every block of data written to that virtual disk. Upon a write, these message digests are consulted in order to determine if a particular block of data has already been written, if so, it is not written again, and if not, it is written. All de-duplication virtual disks are written to a single system virtual disk within the storage platform. De-duplication occurs over the entire storage platform and over all its virtual disks because all message digests are consulted before a write is performed for any virtual disk. A read for a de-duplication virtual desk reads from the system virtual disk.
1. A system comprising:
a data storage platform in communication with a computer server, wherein the data storage platform comprises computer nodes, and wherein each computer node comprises a computer and storage devices; and
the computer server, which is configured to execute a controller virtual machine, which is configured to intercept write requests issued by software applications hosted by the computer server, and wherein each write request is addressed to one of a plurality of virtual disks configured on the data storage platform;
wherein a system deduplication virtual disk, which is distinct from the plurality of virtual disks, is configured on the data storage platform, is partitioned across some of the computer nodes, and is not accessible as a storage target to the software applications hosted by the computer server;
wherein the controller virtual machine executing at the computer server is configured to:
receive from a software application executing at the computer server, a write request to write data onto a first virtual disk among the plurality of virtual disks at the data storage platform, wherein the first virtual disk is configured with deduplication enabled,
calculate a hash value for each block within the data in the write request,
determine whether each hash value is present in a table that includes hash values of blocks that have been written to the system deduplication virtual disk in the data storage platform,
based on the controller virtual machine determining that the table comprises a first hash value calculated for a first block within the data in the write request, (i) retrieve from the table a first offset indicating where a given block having the first hash value is stored in the system deduplication virtual disk, and (ii) write an identifier of the given block and the first offset to metadata in the data storage platform, wherein the metadata is associated with the first virtual disk, wherein the first block is not written to the first virtual disk indicated in the write request,
based on the controller virtual machine determining that the table does not comprise a second hash value calculated for a second block within the data in the write request, (iii) write the second block to the system deduplication virtual disk at a second offset, and (iv) write an identifier of the second block and the second offset to metadata in the data storage platform associated with the first virtual disk, wherein the second block is not written to the first virtual disk indicated in the write request; and
wherein the controller virtual machine executing at the computer server is further configured to: determine whether each hash value is present in the table only if the first virtual disk is configured with deduplication enabled.
2. The system of claim 1 , wherein the system deduplication virtual disk stores blocks for all virtual disks among the plurality of virtual disks at the data storage platform that are configured with deduplication enabled.
3. The system of claim 1 , wherein the table that includes hash values of blocks that have been written to the system deduplication virtual disk in the data storage platform is stored at and maintained by a metadata module of each computer node.
4. The system of claim 3 , wherein at least part of the table that includes hash values of blocks that have been written to the system deduplication virtual disk in the data storage platform is additionally stored at a cache of the controller virtual machine.
5. The system of claim 1 , wherein the table that includes hash values of blocks that have been written to the system deduplication virtual disk in the data storage platform is stored at and maintained by a metadata module of each computer node;
wherein part of the table is additionally stored at a cache of the controller virtual machine; and
wherein when the controller virtual machine determines that the part of the table does not comprise a given hash value, the controller virtual machine is further configured to contact a metadata module at a computer node to determine whether the table at the metadata module comprises the given hash value.
6. The system of claim 1 , wherein the table that includes hash values of blocks that have been written to the system deduplication virtual disk in the data storage platform is stored at and maintained by a metadata module of each computer node;
wherein the table is additionally stored at a cache associated with the controller virtual machine at the computer server; and
wherein to determine whether the table comprises a given hash value, the controller virtual machine checks the table stored at the cache associated with the controller virtual machine.
7. The system of claim 1 , wherein blocks are written sequentially to the system deduplication virtual disk.
8. The system of claim 1 , wherein blocks are written sequentially to the system deduplication virtual disk; and
wherein the controller virtual machine periodically requests, from the data storage platform, information about next available offsets available in the system deduplication virtual disk.
9. The system of claim 1 , wherein blocks are written sequentially to the system deduplication virtual disk; and
wherein the controller virtual machine periodically requests, from the data storage platform, information about next available offsets available in the system deduplication virtual disk, and wherein the information is available from one or more metadata modules of the data storage platform.
10. The system of claim 1 , wherein write operations do not overwrite older versions of blocks, and wherein data is stored at the data storage platform in chunks, each chunk having a name that identifies a version of the data in the chunk.
11. The system of claim 1 , wherein the controller virtual machine is further configured to:
receive from a software application executing at the computer server, a read request to read a third block from a third offset of the first virtual disk at the data storage platform, which is configured with deduplication enabled,
determine a storage container representing a partition of the system deduplication virtual disk comprising a data block corresponding to the third offset of the first virtual disk,
determine a computer node hosting the storage container,
read the data block from the storage container at the computer node hosting the storage container, and
return the data block read from the storage container to the software application that issued the read request at the computer server.
12. The system of claim 1 , wherein the controller virtual machine is further configured to:
receive from a software application executing at the computer server, a read request to read a third block from a third offset of the first virtual disk at the data storage platform, which is configured with deduplication enabled,
based on determining that a cache at the computer server comprises the third block, wherein the cache is associated with the controller virtual machine, return the third block read from the cache to the software application that issued the read request at the computer server.
13. A computer-implemented method comprising:
by a controller virtual machine that is hosted by a computer server, which is distinct from computer nodes of a data storage platform, intercepting write requests issued by software applications hosted by the computer server,
wherein each of the computer nodes comprises a computer and storage devices,
wherein each write request is addressed to one of a plurality of virtual disks configured on the data storage platform,
wherein a system deduplication virtual disk, which is configured on the data storage platform, is partitioned across some of the computer nodes, and is not accessible as a storage target to the software applications hosted by the computer server;
by the controller virtual machine:
receiving from a software application executing at the computer server, a write request to write data onto a first virtual disk among the plurality of virtual disks at the data storage platform, wherein the first virtual disk is provisioned with deduplication enabled,
calculating a hash value for each block within the data in the write request, and
determining whether each hash value is present in a table that includes hash values of blocks that have been written to the system deduplication virtual disk in the data storage platform;
based on the controller virtual machine determining that the table comprises a first hash value calculated for a first block within the data in the write request, (i) retrieving from the table a first offset indicating where a data block having the first hash value is stored in the system deduplication virtual disk, and (ii) writing an identifier of the data block and the first offset to metadata in the data storage platform that is associated with the first virtual disk, wherein the first block is not written to the first virtual disk indicated in the write request;
based on the controller virtual machine determining that the table does not comprise a second hash value calculated for a second block within the data in the write request, (iii) writing the second block to the system deduplication virtual disk at a second offset, and (iv) writing an identifier of the second block and the second offset to metadata in the data storage platform associated with the first virtual disk, wherein the second block is not written to the first virtual disk indicated in the write request; and
determining whether each hash value is present in the table only if the first virtual disk is provisioned with deduplication enabled.
14. The method of claim 13 , wherein the system deduplication virtual disk stores blocks for all virtual disks among the plurality of virtual disks at the data storage platform that are provisioned with deduplication enabled.
15. The method of claim 13 , wherein the table that includes hash values of blocks that have been written to the system deduplication virtual disk in the data storage platform is stored at and maintained by a metadata module of one or more of the computer nodes.
16. The method of claim 15 , wherein at least part of the table that includes hash values of blocks that have been written to the system deduplication virtual disk in the data storage platform is additionally stored at a cache of the controller virtual machine.
17. The method of claim 13 , wherein the table that includes hash values of blocks that have been written to the system deduplication virtual disk in the data storage platform is stored at and maintained by a metadata module of one or more of the computer nodes;
wherein part of the table is additionally stored at a cache of the controller virtual machine; and
wherein when the controller virtual machine determines that the part of the table does not comprise a given hash value, the controller virtual machine contacts a metadata module to determine whether the table at the metadata module comprises the given hash value.
18. The method of claim 13 , wherein the table that includes hash values of blocks that have been written to the system deduplication virtual disk in the data storage platform is stored at and maintained by a metadata module of one or more of the computer nodes;
wherein the table is additionally stored at a cache of the controller virtual machine; and
wherein to determine whether the table comprises a given hash value, the controller virtual machine checks the table stored at the cache of the controller virtual machine.
19. The method of claim 13 further comprising:
by the controller virtual machine:
receiving from a software application executing at the computer server, a read request to read a third block from a third offset of the first virtual disk at the data storage platform, which is configured with deduplication enabled,
determining a storage container representing a partition of the system deduplication virtual disk comprising a data block corresponding to the third offset of the first virtual disk,
determining a computer node hosting the storage container,
reading the data block from the storage container at the computer node hosting the storage container, and
returning the data block read from the storage container to the software application that issued the read request for the third block at the computer server.
20. The method of claim 13 further comprising:
by the controller virtual machine:
receiving from a software application executing at the computer server, a read request to read a third block from a third offset of the first virtual disk at the data storage platform, which is configured with deduplication enabled, and
determining that a cache at the computer server comprises the third block, wherein the cache is associated with the controller virtual machine, and
without communicating further with the data storage platform, returning the third block read from the cache to the software application that issued the read request for the third block at the computer server.