Utilizing fixed-sized and variable-length data chunks to perform source side deduplication
Range information associated with one or more objects is received from a storage system. One or more missing ranges and/or one or more mismatched ranges associated with the one or more objects is determined based on the received range information. A plurality of data chunk identifiers associated with a plurality of variable-length data chunks included in the one or more determined ranges associated with the one or more objects is provided to the storage system. A response that is used to identify among the plurality of variable-length data chunks, one or more variable-length data chunks not already stored in a storage associated with the storage system is received from the storage system. Content of the identified one or more variable-length data chunks is provided to the storage system.
1 . A method, comprising:
receiving, by a backup agent executed at a source computing system and from a storage system separate from the source computing system, range information associated with one or more objects, the range information including a plurality of range chunk identifiers, wherein each of the plurality of range chunk identifiers identifies a corresponding range of chunks within a corresponding object of the one or more objects stored as one or more fixed-length data bricks, wherein each of the one or more fixed-length data bricks includes one or more variable-length data chunks identified in the range of chunks, and each of the fixed-length data bricks is associated with brick metadata that includes a respective one of the plurality of range chunk identifiers;
based on the plurality of range chunk identifiers included in the range information received from the storage system, determining, by the backup agent executed at the source computing system, one or more missing ranges or one or more mismatched ranges associated with the one or more objects, wherein:
determining the one or more mismatched ranges includes comparing a range chunk identifier from the plurality of range chunk identifiers that corresponds to a portion of the one or more objects to a current range chunk identifier that corresponds to the portion of the one or more objects,
a mismatched range of the one or more mismatched ranges exists when the range chunk identifier identifies a different range of data chunks than a range of data chunks of the current range chunk identifier, and
a missing range of the one or more missing ranges exists when the range information does not cover a range corresponding to the current range chunk identifier;
providing, by the backup agent executing at the source computing system and to the storage system, a plurality of data chunk identifiers associated with the one or more variable-length data chunks included in the at least one of the one or more missing ranges or the one or more mismatched ranges associated with the one or more objects;
receiving, by the backup agent executed at the source computing system and from the storage system, a response that is used to identify, among the plurality of variable-length data chunks, one or more variable-length data chunks not already stored in a storage associated with the storage system; and
providing, by the backup agent executed at the source computing system and to the storage system, content of the one or more variable-length data chunks.
2 . The method of claim 1 , wherein each of the plurality of range chunk identifiers corresponds to a particular range of the corresponding object.
3 . The method of claim 1 , wherein the missing range of the one or more missing ranges corresponds to a portion of an object of the one or more objects that was not previously backed up.
4 . The method of claim 1 , wherein determining the one or more missing ranges associated with the one or more objects includes determining, by the backup agent executed at the source computing system, that the range information received from the storage system does not cover the range corresponding to the current range chunk identifier.
5 . The method of claim 1 , further comprising chunking, by the computing system, data associated with the one or more missing ranges or the one or more mismatched ranges into the plurality of variable-length data chunks.
6 . The method of claim 1 , further comprising generating, by the backup agent executed at the computing system, the plurality of data chunk identifiers associated with the plurality of variable-length data chunks.
7 . The method of claim 1 , wherein the storage system compares the plurality of data chunk identifiers associated with the plurality of variable-length chunks to a second plurality of data chunk identifiers included in a chunk metadata data structure that identifies a plurality of data chunks stored in the storage associated with the storage system.
8 . The method of claim 7 , wherein in response to a determination that a data chunk identifier associated with a variable-length data chunk is not included in the chunk metadata data structure, the storage system includes the data chunk identifier associated with the variable-length data chunk in the response.
9 . The method of claim 8 , wherein in response to a determination that a data chunk identifier associated with a variable-length data chunk is included in the chunk metadata data structure, the storage system excludes the data chunk identifier associated with the variable-length data chunk from the response.
10 . Non-transitory computer readable media storing computer instructions that, when executed by a processor of a source computing system, cause the processor to execute a backup agent to:
receive, from a storage system, range information associated with one or more objects, the range information including a plurality of range chunk identifiers, wherein each of the plurality of range chunk identifiers identifies a corresponding range of chunks within a corresponding object of the one or more objects stored as one or more fixed-length data bricks, wherein each of the one or more fixed-length data bricks includes one or more variable-length data chunks identified in the range of chunks, and each of the fixed-length data bricks is associated with brick metadata that includes a respective one of the plurality of range chunk identifiers;
based on the plurality of range chunk identifiers included in the range information received from the storage system, determine at least one of one or more missing ranges or one or more mismatched ranges associated with the one or more objects, wherein:
determining the one or more mismatched ranges includes comparing a range chunk identifier from the plurality of range chunk identifiers that corresponds to a portion of the one or more objects to a current range chunk identifier that corresponds to the portion of the one or more objects,
a mismatched range of the one or more mismatched ranges exists when the range chunk identifier identifies a different range of data chunks than a range of data chunks of the current range chunk identifier, and
a missing range of the one or more missing ranges exists when the range information does not cover a range corresponding to the current range chunk identifier;
provide, to the storage system, a plurality of data chunk identifiers associated with the one or more variable-length data chunks included in the at least one of the one or more missing ranges or the one or more mismatched ranges associated with the one or more objects;
receive, from the storage system, a response that is used to identify, among the one or more of variable-length data chunks, one or more variable-length data chunks not already stored in a storage associated with the storage system; and
provide, to the storage system, content of the one or more variable-length data chunks.
11 . The non-transitory computer readable media of claim 10 , wherein each of the plurality of range chunk identifiers corresponds to a particular range of the corresponding object.
12 . The non-transitory computer readable media of claim 10 , wherein to determine the one or more missing ranges associated with the one or more objects, the computer instructions cause the processor to determine that the range information received from the storage system does not cover the range corresponding to the current range chunk identifier.
13 . The non-transitory computer readable media of claim 10 , wherein the computer instructions cause the processor to chunk data associated with the one or more missing ranges or the one or more mismatched ranges into the plurality of variable-length data chunks.
14 . The non-transitory computer readable media of claim 10 , wherein the computer instructions cause the processor to generate the plurality of data chunk identifiers associated with the plurality of variable-length data chunks.
15 . A system, comprising:
a processor of a source computing system configured to execute a backup agent to:
receive, from a storage system, range information associated with one or more objects, the range information including a plurality of range chunk identifiers, wherein each of the plurality of range chunk identifiers identifies a corresponding range of chunks within a corresponding object the one or more objects stored as one or more fixed-length data bricks, wherein each of the one or more fixed-length data bricks includes one or more variable-length data chunks identified in the range of chunks, and each of the fixed-length data bricks is associated with brick metadata that includes a respective one of the plurality of range chunk identifiers;
based on the plurality of range chunk identifiers included in the range information received from the storage system, determine at least one of one or more missing ranges or one or more mismatched ranges associated with the one or more objects, wherein:
determining the one or more mismatched ranges includes comparing a range chunk identifier from the plurality of range chunk identifiers that corresponds to a portion of the one or more objects to a current range chunk identifier that corresponds to the portion of the one or more objects,
a mismatched range of the one or more mismatched ranges exists when the range chunk identifier identifies a different range of data chunks than a range of data chunks of the current range chunk identifier, and
a missing range of the one or more missing ranges exists when the range information does not cover a range corresponding to the current range chunk identifier;
provide, to the storage system, a plurality of data chunk identifiers associated with the one or more variable-length data chunks included in the at least one of the one or more missing ranges or the one or more mismatched ranges associated with the one or more objects;
receive, from the storage system, a response that is used to identify among the plurality of variable-length data chunks, one or more variable-length data chunks not already stored in a storage associated with the storage system; and
provide, to the storage system, content of the one or more variable-length data chunks; and
a memory coupled to the processor and configured to provide the processor with instructions for executing the backup agent.