Systems and methods of chunking data for secure data storage across multiple cloud providers
Techniques for chunking data in data storage systems that provide increased data storage security across multiple cloud storage providers. The techniques employ a chunking engine and a policy engine, which evaluates one or more storage policies relating to, for example, cost, security, and/or network conditions in view of services and/or requirements of the multiple cloud storage providers. Having evaluated such storage policies, the policy engine generates and provides operating parameters to the chunking engine, which uses the operating parameters when chunking and/or distributing the data across the multiple cloud storage providers, thereby satisfying the respective storage policies. In this way, users of data storage systems obtain the benefits of cloud storage resources and/or services while reducing their data security concern and optimizing the total cost of data storage.
1. A method of chunking data in a data storage system that provides increased data storage security across multiple cloud storage providers, comprising:
receiving, at the data storage system from a host computer, a file storage request pertaining to a file, the data storage system including at least a policy engine and a chunking engine;
evaluating, by the policy engine, a storage policy of the data storage system to determine whether one or more cloud storage parameters of the multiple cloud storage providers at least partially conform to the storage policy;
having determined that the one or more cloud storage parameters at least partially conform to the storage policy, determining, based on the one or more cloud storage parameters, multiple levels of conformity of the multiple cloud storage providers to the storage policy, the multiple levels of conformity including at least a best level of conformity to the storage policy and a next best level of conformity to the storage policy;
generating, by the policy engine, one or more operating parameters of the chunking engine for implementing the storage policy at the multiple cloud storage providers, the one or more operating parameters specifying at least one chunk size;
partitioning, by the chunking engine, the file into a plurality of chunks having the at least one chunk size;
transmitting the plurality of chunks having the at least one chunk size for storage across the respective multiple cloud storage providers having the best level of conformity to the storage policy;
determining that a first cloud storage provider from among the multiple cloud storage providers having the best level of conformity to the storage policy has one or more of a network failure, a server failure, and a breach in security; and
transmitting at least some of the plurality of chunks originally destined for storage at the first cloud storage provider to a second cloud storage provider from among the multiple cloud storage providers having the next best level of conformity to the storage policy.
2. The method of claim 1 wherein the generating of the one or more operating parameters of the chunking engine includes determining a fixed size of a chunk, and wherein the partitioning of the file includes partitioning the file into the plurality of chunks, each of the plurality of chunks having the fixed size.
3. The method of claim 1 wherein the generating of the one or more operating parameters of the chunking engine includes determining a variable size of a chunk, and wherein the partitioning of the file includes partitioning the file into the plurality of chunks, at least some of the plurality of chunks having the variable size.
4. The method of claim 1 further comprising:
having transmitted the plurality of chunks for storage across the multiple cloud storage providers, generating metadata information for locating and accessing the plurality of chunks, the metadata information including at least some of (i) identifiers (IDs) of the multiple cloud storage providers, (ii) the at least one chunk size of the plurality of chunks, (iii) an identifier (ID) of each respective chunk, (iv) a location of each chunk across the multiple cloud storage providers, (v) a signature of each chunk, (vi) an overall size of the file made up of the plurality of chunks, (vii) a checksum of the file, and (viii) reconstruction information for the file using the plurality of chunks.
5. The method of claim 4 further comprising:
retrieving, by the data storage system, the plurality of chunks from across the multiple cloud storage providers using at least some of the metadata information pertaining to the plurality of chunks.
6. The method of claim 5 further comprising:
having retrieved the plurality of chunks from across the multiple cloud storage providers, reconstructing the file using at least some of the metadata information pertaining to the file.
7. The method of claim 1 wherein the evaluating of the storage policy of the data storage system includes determining that portions of the file are to be stored at one or more of the multiple cloud storage providers that satisfy one or more of a predetermined network bandwidth and a predetermined network capacity.
8. A data storage system that chunks data in a manner providing increased data storage security across multiple cloud storage providers, comprising:
a memory storing a policy engine and a chunking engine; and
storage processing circuitry,
wherein the storage processing circuitry is configured to execute the policy engine:
to evaluate a storage policy of the data storage system to
determine whether one or more cloud storage parameters of the multiple cloud storage providers at least partially conform to the storage policy;
having determined that the one or more cloud storage parameters at least partially conform to the storage policy, to determine, based on the one or more cloud storage parameters, multiple levels of conformity of the multiple cloud storage providers to the storage policy, the multiple levels of conformity including at least a best level of conformity to the storage policy and a next best level of conformity to the storage policy;
to generate one or more operating parameters of the chunking engine for implementing the storage policy at the multiple cloud storage providers, the one or more operating parameters specifying at least one chunk size,
wherein the storage processing circuitry is further configured to execute the chunking engine to partition a file into a plurality of chunks having the at least one chunk size, and
wherein the data storage system is configured to transmit the plurality of chunks having the at least one chunk size for storage across the multiple cloud storage providers having the best level of conformity to the storage policy; and
to determine that a first cloud storage provider from among the multiple cloud storage providers having the best level of conformity to the storage policy has one or more of a network failure, a server failure, and a breach in security,
wherein the data storage system is further configured to transmit at least some of the plurality of chunks originally destined for storage at the first cloud storage provider to a second cloud storage provider from among the multiple cloud storage providers having the next best level of conformity to the storage policy.
9. The data storage system of claim 8 wherein the storage processing circuitry is further configured to execute the policy engine to determine a fixed size of a chunk, and wherein the storage processing circuitry is further configured to execute the chunking engine to partition the file into the plurality of chunks, each of the plurality of chunks having the fixed size.
10. The data storage system of claim 8 wherein the storage processing circuitry is further configured to execute the policy engine to determine a variable size of a chunk, and wherein the storage processing circuitry is further configured to execute the chunking engine to partition the file into the plurality of chunks, at least some of the plurality of chunks having the variable size.
11. The data storage system of claim 8 wherein the storage processing circuitry is further configured to execute the chunking engine, once the plurality of chunks are transmitted for storage across the multiple cloud storage providers, to generate metadata information for locating and accessing the plurality of chunks, the metadata information including at least some of (i) identifiers (IDs) of the multiple cloud storage providers, (ii) the at least one chunk size of the plurality of chunks, (iii) an identifier (ID) of each respective chunk, (iv) a location of each chunk across the multiple cloud storage providers, (v) a signature of each chunk, (vi) an overall size of the file made up of the plurality of chunks, (vii) a checksum of the file, and (viii) reconstruction information for the file using the plurality of chunks.
12. The data storage system of claim 11 wherein the data storage system is further configured to retrieve the plurality of chunks from across the multiple cloud storage providers using at least some of the metadata information pertaining to the plurality of chunks.
13. The data storage system of claim 12 wherein the data storage system is further configured, having retrieved the plurality of chunks from across the multiple cloud storage providers, to reconstruct the file using at least some of the metadata information pertaining to the file.
14. A computer program product having a non-transitory computer readable medium that stores a set of instructions that, when carried out by computerized circuitry, cause the computerized circuitry to perform a method of chunking data in a data storage system that provides increased data storage security across multiple cloud storage providers, the method comprising:
receiving, at the data storage system from a host computer, a file storage request pertaining to a file, the data storage system including at least a policy engine and a chunking engine;
evaluating, by the policy engine, a storage policy of the data storage system to determine whether one or more cloud storage parameters of the multiple cloud storage providers at least partially conform to the storage policy;
having determined that the one or more cloud storage parameters at least partially conform to the storage policy, determining, based on the one or more cloud storage parameters, multiple levels of conformity of the multiple cloud storage providers to the storage policy, the multiple levels of conformity including at least a best level of conformity to the storage policy and a next best level of conformity to the storage policy;
generating, by the policy engine, one or more operating parameters of the chunking engine for implementing the storage policy at the multiple cloud storage providers, the one or more operating parameters specifying at least one chunk size;
partitioning, by the chunking engine, the file into a plurality of chunks having the at least one chunk size;
transmitting the plurality of chunks having the at least one chunk size for storage across the multiple cloud storage providers having the best level of conformity to the storage policy;
determining that a first cloud storage provider from among the multiple cloud storage providers having the best level of conformity to the storage policy has one or more of a network failure, a server failure, and a breach in security; and
transmitting at least some of the plurality of chunks originally destined for storage at the first cloud storage provider to a second cloud storage provider from among the multiple cloud storage providers having the next best level of conformity to the storage policy.
15. The computer program product of claim 14 wherein the generating of the one or more operating parameters of the chunking engine includes determining a variable size of a chunk, and wherein the partitioning of the file includes partitioning the file into the plurality of chunks, at least some of the plurality of chunks having the variable size.