LARGE OBJECT PACKING FOR STORAGE EFFICIENCY
One example method includes receiving data, partitioning the data according to their respective similarity groups, and the similarity groups collectively define a range of similarity groups, deduplicating the data after the partitioning, packing unique data segments remaining after deduplicating into one or more compression regions, compressing the compression regions, and writing an object, that includes the compression regions, to a durable log. The deduplicating and compressing for a similarity group may be performed by a dedup-compression instances uniquely assigned to that similarity group.
1 . A method, comprising the operations:
receiving data;
partitioning the data according to their respective similarity groups, and the similarity groups collectively define a range of similarity groups;
deduplicating the data after the partitioning;
packing unique data segments remaining after deduplicating into one or more compression regions;
compressing the compression regions; and
writing an object, that includes the compression regions, to a durable log.
2 . The method as recited in claim 1 , wherein the object includes one or more containers, and one of the containers includes the compression regions.
3 . The method as recited in claim 1 , wherein the operations further comprise combining the compression regions into a container, and combining the container with one or more additional containers to form the object.
4 . The method as recited in claim 1 , wherein a size of a container that is included in the object is adjusted based on a locality of data stored in the container.
5 . The method as recited in claim 1 , wherein a respective dedup-compress instance is assigned to each of the similarity groups in the range of similarity groups, and only the respective dedup-compress instance performs deduplication and compression for the similarity group to which that dedup-compress instance is assigned.
6 . The method as recited in claim 1 , wherein all data segments in the compression regions and the object come from the same similarity group.
7 . The method as recited in claim 1 , wherein the object is accessible at the log even in the event of a system failure.
8 . The method as recited in claim 1 , wherein the object is moved to object storage at some point after being written to the durable log, and the object storage has a higher latency for read and write operations than a latency of the durable log for read and write operations.
9 . The method as recited in claim 1 , wherein the packing is performed by a packer module that is an element of a dedup-compress instance that performs the deduplicating and the compressing.
10 . The method as recited in claim 1 , wherein the unique data segments in a compression region are consecutively written and are represented with a set of fingerprints that are loadable with one storage I/O to a cache prior to deduplication.
11 . A non-transitory storage medium having stored therein instructions that are executable by one or more hardware processors to perform operations comprising:
receiving data;
partitioning the data according to their respective similarity groups, and the similarity groups collectively define a range of similarity groups;
deduplicating the data after the partitioning;
packing unique data segments remaining after deduplicating into one or more compression regions;
compressing the compression regions; and
writing an object, that includes the compression regions, to a durable log.
12 . The non-transitory storage medium as recited in claim 11 , wherein the object includes one or more containers, and one of the containers includes the compression regions.
13 . The non-transitory storage medium as recited in claim 11 , wherein the operations further comprise combining the compression regions into a container, and combining the container with one or more additional containers to form the object.
14 . The non-transitory storage medium as recited in claim 11 , wherein a size of a container that is included in the object is adjusted based on a locality of data stored in the container.
15 . The non-transitory storage medium as recited in claim 11 , wherein a respective dedup-compress instance is assigned to each of the similarity groups in the range of similarity groups, and only the respective dedup-compress instance performs deduplication and compression for the similarity group to which that dedup-compress instance is assigned.
16 . The non-transitory storage medium as recited in claim 11 , wherein all data segments in the compression regions and the object come from the same similarity group.
17 . The non-transitory storage medium as recited in claim 11 , wherein the object is accessible at the log even in the event of a system failure.
18 . The non-transitory storage medium as recited in claim 11 , wherein the object is moved to object storage at some point after being written to the durable log, and the object storage has a higher latency for read and write operations than a latency of the durable log for read and write operations.
19 . The non-transitory storage medium as recited in claim 11 , wherein the packing is performed by a packer module that is an element of a dedup-compress instance that performs the deduplicating and the compressing.
20 . The non-transitory storage medium as recited in claim 11 , wherein the unique data segments in a compression region are consecutively written and are represented with a set of fingerprints that are loadable with one storage I/O to a cache prior to deduplication.