Data reduction in a cloud-based storage system
Data reduction across different cloud-based storage systems, including: performing one or more global data reduction processes on the data using one or more global databases that represent the data associated with the plurality of cloud-based virtual storage systems.
1. A system comprising:
a plurality of cloud-based virtual storage systems implementing one or more virtual controllers, wherein the virtual storage systems store data in one or more backend storage resources; and
one or more virtual drive servers that include a plurality of virtual drives having local instance stores;
a global metadata reference database storing references to the data, wherein the one or more virtual drive servers can access the global metadata reference database;
the system being configured to:
perform one or more global data reduction processes on the data using one or more global databases that represent the data associated with the plurality of cloud-based virtual storage systems, wherein the one or more global data reduction processes comprise determining that a data element in the global metadata reference database is associated with two or more virtual drive servers.
2. The system of claim 1 , wherein the system is further configured to perform the one or more global data reduction processes using the one or more virtual drive servers.
3. The system of claim 1 , wherein the data includes two or more data elements whose similarity to each other satisfies a similarity threshold, wherein the two or more data elements are stored by different virtual storage systems.
4. The system of claim 1 , wherein performing the one or more global data reduction processes includes performing a global garbage collection process, comprising:
in response to determining that the data element in the global metadata reference database is associated with two or more virtual drive servers, delaying the global garbage collection process until at least one of the two or more virtual drive servers has issued an instruction to store the data element at a different location that a previous storage location.
5. The system of claim 4 , wherein performing the global garbage collection process further comprises:
storing an identifier of a virtual drive server of the two or more virtual drive servers;
determining that an amount of data to be garbage-collected from a data structure of the one or more backend storage resources satisfies a threshold; and
based on the determination:
migrating at least a portion of the data stored by the data structure to another data structure; and
garbage-collecting the data structure.
6. The system of claim 4 , wherein the one or more global databases includes a global fingerprint database storing data fingerprints for the data, and wherein performing the one or more global data reduction processes further comprises:
determining, using the global fingerprint database, that new data received via an I/O operation was previously stored in one or more of the backend storage resources; and
based on the determination:
storing a reference to the new data in the global metadata reference database; and
preventing storage of the new data in one or more of the backend storage resources.
7. The system of claim 6 , wherein the system comprises:
determining that a first portion of the new data was not previously stored in one or more of the backend storage resources; and
based on the determination:
storing the first portion of the new data in one or more of the backend storage resources; and
storing a reference to a remainder of the new data excluding the first portion in the global metadata reference database.
8. The system of claim 1 , wherein the one or more global data reduction processes on the data comprises similarity compression.
9. The system of claim 1 , wherein the system is further configured to perform one or more of deduplication or compression using one or more of the local instance stores before transferring written data to the one or more backend storage resources.
10. The system of claim 9 , wherein the system is further configured to perform one or more of the deduplication or the compression during a process of transferring the written data to the one or more backend storage resources, without writing the data to one or more of the local instance stores.
11. The system of claim 1 , wherein the one or more backend storage resources includes one or more of object-based storage resources or block-based storage resources.
12. A method, comprising:
storing references to data on a global metadata reference database, wherein one or more virtual drive servers can access the global metadata reference database; and
performing one or more global data reduction processes on the data using one or more global databases that represent the data associated with a plurality of cloud-based virtual storage systems, the plurality of cloud-based virtual storage systems implementing one or more virtual controllers, wherein the virtual storage systems store the data in one or more backend storage resources, wherein performing the one or more global reduction processes on the data comprises determining that a data element in the global metadata reference database is associated with two or more virtual drive servers.
13. The method of claim 12 , further comprising:
performing the one or more global data reduction processes using the one or more virtual drive servers.
14. The method of claim 12 , wherein the data includes two or more data elements whose similarity to each other satisfies a similarity threshold, wherein the two or more data elements are stored by different virtual storage systems.
15. The method of claim 12 , further comprising:
performing a global garbage collection process using a global metadata reference database storing references to the data, further comprising:
in response to determining that the data element in the global metadata reference database is associated with two or more virtual drive servers, delaying the global garbage collection process until at least one of the two or more virtual drive servers has issued an instruction to store the data element at a different location than a previous storage location.
16. The method of claim 15 , wherein performing the global garbage collection process further comprises:
storing an identifier of a virtual drive server of the two or more virtual drive servers;
determining that an amount of data to be garbage-collected from a data structure of the one or more backend storage resources satisfies a threshold; and
based on the determination:
migrating at least a portion of the data stored by the data structure to another data structure; and
garbage-collecting the data structure.
17. A non-transitory computer readable storage medium which, when executed, cause a processing device to: store references to data on a global metadata reference database, wherein one or more virtual drive servers can access the global metadata reference database; and
perform one or more global data reduction processes on the data using one or more global databases that represent the data associated with a plurality of cloud-based virtual storage systems, wherein the one or more global data reduction processes comprise determining that a data element in the global metadata reference database is associated with two or more virtual drive servers.
18. The non-transitory computer readable storage medium of claim 17 , wherein the processing device is further configured to perform the one or more global data reduction processes using one or more virtual drive servers.
19. The non-transitory computer readable storage medium of claim 17 , wherein the data includes two or more data elements whose similarity to each other satisfies a similarity threshold, wherein the two or more data elements are stored by different virtual storage systems.
20. The non-transitory computer readable storage medium of claim 17 , wherein the processing device is further configured to:
in response to determining that the data element in the global metadata reference database is associated with two or more virtual drive servers, delay a global garbage collection process until at least one of the two or more virtual drive servers has issued an instruction to store the data element at a different location than a previous storage location.