SIMILARITY-BASED GLOBAL DATA REDUCTION ACROSS VIRTUAL STORAGE SYSTEMS

1 . A system for data reduction across different cloud-based storage systems, the system comprising:

a plurality of cloud-based virtual storage systems implementing one or more virtual controllers, wherein the cloud-based virtual storage systems store data in one or more backend storage resources; and

one or more virtual drive servers that include a plurality of virtual drives having local instance stores;

the system being configured to:

generate, using a global metadata reference database that stores similarity features of data associated with the plurality of cloud-based virtual storage systems, a similarity representation for received data by comparing the received data to previously stored data elements;

perform one or more global data reduction processes on the received data together with the at least one of the previously stored data elements based on the similarity representation to generate reduced data; and

store the reduced data.

2 . The system of claim 1 , wherein the system is further configured to perform the one or more global data reduction processes using the one or more virtual drive servers.

3 . The system of claim 1 , wherein the data includes two or more data elements whose similarity to each other satisfies a similarity threshold, wherein the two or more data elements are stored by different virtual storage systems.

4 . The system of claim 1 , wherein performing the one or more global data reduction processes includes performing a global garbage collection process using a global metadata reference database storing references to the data, and wherein the system is further configured to:

determine that a data element stored in the one or more backend storage resources is referenced by two or more virtual drive servers; and

in response to the determination, delay the global garbage collection process until at least one of the two or more virtual drive servers has issued an instruction to store the data element at a different location than a previous storage location.

5 . The system of claim 4 , wherein performing the global garbage collection process further comprises:

storing an identifier of a virtual drive server of the two or more virtual drive servers;

determining that an amount of data to be garbage-collected from a data structure of the one or more backend storage resources satisfies a threshold; and

based on the determination:

migrating at least a portion of the data stored by the data structure to another data structure; and

garbage-collecting the data structure.

6 . The system of claim 4 , wherein the global metadata reference database includes a global fingerprint database storing data fingerprints for the data, and wherein performing the one or more global data reduction processes further comprises:

determining, using the global fingerprint database, that new data received via an I/O operation was previously stored in the one or more backend storage resources; and

based on the determination:

storing a reference to the new data in the global metadata reference database; and

preventing storage of the new data in the one or more backend storage resources.

7 . The system of claim 6 , wherein the system is further configured to:

determine that a first portion of the new data was not previously stored in the one or more backend storage resources; and

based on the determination:

store the first portion of the new data in the one or more backend storage resources; and

store a reference to a remainder of the new data excluding the first portion in the global metadata reference database.

8 . The system of claim 6 , wherein the global metadata reference database includes a global compression table that stores compression metadata for the data, and wherein performing the one or more global data reduction processes further comprises:

determining, using the global compression table, that new data being received via an I/O operation is similar to one or more previously received data elements;

based on the determination:

compressing the new data with the one or more previously received data elements; and

based on the compressing, storing a compression result in the one or more backend storage resources.

9 . The system of claim 1 , wherein the system is further configured to perform one or more of deduplication or compression using one or more of the local instance stores before transferring written data to the one or more backend storage resources.

10 . The system of claim 9 , wherein the system is further configured to perform one or more of the deduplication or the compression during a process of transferring the written data to the one or more backend storage resources, without writing the data to one or more of the local instance stores.

11 . The system of claim 1 , wherein the one or more backend storage resources includes one or more of object-based storage resources or block-based storage resources.

12 . A method comprising:

receiving data at a virtual drive server;

generating, using a global metadata reference database that stores similarity features of data associated with a plurality of cloud-based virtual storage systems, a similarity representation for the received data by comparing the received data to previously stored data elements;

performing one or more global data reduction processes on the received data together with the at least one of the previously stored data elements based on the similarity representation to generate reduced data; and

storing the reduced data in one or more backend storage resources.

13 . The method of claim 12 , wherein performing the one or more global data reduction processes comprises deduplicating the received data or compressing the received data using a local instance store of the virtual drive server.

14 . The method of claim 12 , further comprising:

determining, using a global fingerprint database, that new data has previously been stored in the one or more backend storage resources; and

storing a reference to the new data in the global metadata reference database in response to the determining.

15 . The method of claim 14 , further comprising:

identifying a first portion of the new data that has not been previously stored; and

storing the first portion in the one or more backend storage resources and storing a reference to a remainder of the data.

16 . The method of claim 12 , wherein performing the one or more global data reduction processes comprises:

detecting, using a global compression table, similarity between new data and previously stored data; and

compressing the new data with the previously stored data to generate a compression result for storage.

17 . A non-transitory computer readable storage medium storing instructions which, when executed, cause a processing device to:

receive data at a virtual drive server;

generate, using a global metadata reference database that stores similarity features of data associated with a plurality of cloud-based virtual storage systems, a similarity representation for the received data by comparing the received data to previously stored data elements;

perform one or more global data reduction processes on the received data together with the at least one of the previously stored data elements based on the similarity representation to generate reduced data; and

store the reduced data in one or more backend storage resources.

18 . The non-transitory computer readable storage medium of claim 17 , wherein the instructions further cause the processing device to deduplicate or compress the received data using a local instance store prior to transfer to the one or more backend storage resources.

19 . The non-transitory computer readable storage medium of claim 17 , wherein the instructions further cause the processing device to:

identify that new data that is received was previously stored using a global fingerprint database; and

store a reference to the new data without storing the data itself.

20 . The non-transitory computer readable storage medium of claim 17 , wherein the instructions further cause the processing device to:

determine similarity between new data and previously stored data using a global compression table; and

compress the new data with the previously stored data and store a compression result.