Efficiently storing data in a cloud storage
A specification of content to be stored in a cloud storage is received at a client-side component. A first portion of the content is divided into a plurality of data chunks. One or more data chunks of the plurality of data chunks that are to be sent via a network to be stored in the cloud storage are identified. It is determined whether a batch size of the one or more identified data chunks does not meets a threshold size. One or more data chunks of a second portion of the content that are to be stored in the cloud storage are identified. It is determined that a size of a second batch of data chunks that includes the one or more identified data chunks of the first portion of the content and the one or more identified data chunks of the second portion of the content does not meet the threshold size. It is determined that a batch period is greater than or equal to a batch threshold period. The second batch of data chunks is written to a storage of a cloud server included in a data plane.
1. A method, comprising:
receiving at a client-side component, a specification of content to be stored in a cloud storage;
dividing a first portion of the content into a plurality of data chunks;
identifying one or more data chunks of the plurality of data chunks that are to be sent via a network to be stored in the cloud storage;
determining that a batch size of a first batch of data chunks that includes the one or more identified data chunks does not meet a threshold size for a cloud storage element object to which the one or more identified data chunks are to be written;
in response to a determination that the batch size of the first batch of data chunks does not meet the threshold size for the cloud storage element object to which the first batch of data chunks are to be written, identifying one or more data chunks of a second portion of the content that are to be stored in the cloud storage;
determining that a size of a second batch of data chunks that includes the one or more identified data chunks of the first portion of the content and the one or more identified data chunks of the second portion of the content does not meet the threshold size;
determining that a batch period is greater than or equal to a batch threshold period; and
in response to determining that the size of the second batch of data chunks does not meet the threshold size and determining that the batch period is greater than or equal to the batch threshold period, writing the second batch of data chunks to a storage of a cloud server included in a data plane, wherein the storage of the cloud server included in the data plane is different than a cloud storage tier for a batch of data chunks having the batch size that meets the threshold size.
2. The method of claim 1 , further comprising receiving the first portion of the content from a source system.
3. The method of claim 1 , wherein identifying the one or more data chunks of the plurality of data chunks that are to be sent via the network to be stored in the cloud storage includes:
computing one or more corresponding chunk identifiers for each of the data chunks of the plurality of data chunks;
sending the one or more corresponding chunk identifiers to a file system manager at the data plane, wherein the file system manager compares the one or more corresponding chunk identifiers to chunk identifiers included in a deduplication table; and
receiving from the file system manager an indication of one or more chunk identifiers that are not included in the deduplication table.
4. The method of claim 1 , wherein the cloud server aggregates the second batch of data chunks with one or more other batches of data chunks.
5. The method of claim 4 , wherein the one or more other batches of data chunks are received from the client-side component.
6. The method of claim 4 , wherein at least one of the one or more other batches of data chunks is received from a second client-side component.
7. The method of claim 6 , wherein the second client-side component is located in a same datacenter as the client-side component.
8. The method of claim 6 , wherein the second client-side component is located in a different datacenter as the client-side component.
9. The method of claim 4 , wherein the cloud server determines that a cumulative size of the aggregated data chunks is greater than an aggregated threshold size and less than a maximum size for the cloud storage element object.
10. The method of claim 9 , wherein in response to determining that the cumulative size of the aggregated data chunks is greater than the aggregated threshold size and less than the maximum size for the cloud storage element object, the cloud server writes the aggregated data chunks to the cloud storage element object.
11. The method of claim 10 , wherein the aggregated data chunks are encrypted.
12. The method of claim 10 , wherein the aggregated data chunks are compressed.
13. The method of claim 1 , further comprising receiving a reference to the cloud storage element object to which the one or more identified data chunks are to be written.
14. The method of claim 13 , wherein the reference includes an identifier of the cloud storage element object.
15. The method of claim 1 , wherein the cloud storage element object is available for a threshold period of time.
16. The method of claim 1 , wherein the cloud storage element object is stored in a first performance tier of the cloud storage.
17. A computer program product embodied in non-transitory computer readable storage media storing instructions that, when executed, cause one or more processors to:
receive at a client-side component, a specification of content to be stored in a cloud storage;
divide a first portion of the content into a plurality of data chunks;
identify one or more data chunks of the plurality of data chunks that are to be sent via a network to be stored in the cloud storage;
determine that a batch size of a first batch of data chunks that includes the one or more identified data chunks does not meet a threshold size for a cloud storage element object to which the one or more identified data chunks are to be written;
in response to a determination that the batch size of the first batch of data chunks does not meet the threshold size for the cloud storage element object to which the first batch of data chunks are to be written, identify one or more data chunks of a second portion of the content that are to be stored in the cloud storage;
determine that a size of a second batch of data chunks that includes the one or more identified data chunks of the first portion of the content and the one or more identified data chunks of the second portion of the content does not meet the threshold size;
determine that a batch period is greater than or equal to a batch threshold period; and
in response to determining that the size of the second batch of data chunks does not meet the threshold size and determining that the batch period is greater than or equal to the batch threshold period, write the second batch of data chunks to a storage of a cloud server included in a data plane, wherein the storage of the cloud server included in the data plane is different than a cloud storage tier for a batch of data chunks having the batch size that meets the threshold size.
18. The computer program product of claim 17 , further comprising instructions that, when executed, cause the one or more processors to receive the first portion of the content from a source system.
19. The computer program product of claim 17 , wherein the instructions that, when executed, cause the one or more processors to identify the one or more data chunks of the plurality of data chunks that are to be sent via the network to be stored in the cloud storage include instructions that, when executed, cause the one or more processors to:
compute one or more corresponding chunk identifiers for each of the data chunks of the plurality of data chunks;
send the one or more corresponding chunk identifiers to a file system manager at the data plane, wherein the file system manager compares the one or more corresponding chunk identifiers to chunk identifiers included in a deduplication table; and
receive from the file system manager an indication of one or more chunk identifiers that are not included in the deduplication table.
20. A system, comprising:
a memory storing instructions; and
one or more processors coupled to the memory and configured to execute the instructions to:
receive a specification of content to be stored in a cloud storage;
divide a first portion of the content into a plurality of data chunks;
identify one or more data chunks of the plurality of data chunks that are to be sent via a network to be stored in the cloud storage;
determine that a batch size of a first batch of data chunks that includes the one or more identified data chunks does not meet a threshold size for a cloud storage element object to which the one or more identified data chunks are to be written;
in response to a determination that the batch size of the first batch of data chunks does not meet the threshold size for cloud storage element object to which the first batch of data chunks are to be written, identifying one or more data chunks of a second portion of the content that are to be stored in the cloud storage;
determine that a size of a second batch of data chunks that includes the one or more identified data chunks of the first portion of the content and the one or more identified data chunks of the second portion of the content does not meet the threshold size;
determine that a batch period is greater than or equal to a batch threshold period; and
in response to determining that the size of the second batch of data chunks does not meet the threshold size and determining that the batch period is greater than or equal to the batch threshold period, writing the second batch of data chunks to a storage of a cloud server included in a data plane, wherein the storage of the cloud server included in the data plane is different than a cloud storage tier for a batch of data chunks having the batch size that meets the threshold size.