Systems and methods for block-data clustering-based deduplication
A method may include calculating a cryptographic hash for each of a plurality of data blocks stored on a storage resource, calculating a locality-preserving hash for each of the plurality of data blocks, and performing clustering based on the locality-preserving hashes. The method may also include, for each particular cluster setting a cluster center data block for the particular cluster and for each particular data block of the particular cluster, storing the particular data block as a cluster member data block including a reference to the cluster center data block for the particular cluster, a list of differences of the particular data block from the cluster member data block, the cryptographic hash for the particular data block to identify the particular data block, and the locality-preserving hash for the particular data block for manipulation of the particular data block.
1. An information handling system comprising:
a processor; and
a deduplication agent embodied in a program of instructions stored in non-transitory computer-readable media, the deduplication agent configured to, when read and executed by the processor:
calculate a cryptographic hash for each of a plurality of data blocks stored on a storage resource;
calculate, with a locality-preserving hash function, a locality-preserving hash for each of the plurality of data blocks wherein a distance between locality-preserving hashes of two inputs that have a first input distance between them is less than a distance between locality-preserving hashes for two inputs that have a second distance between them when the first input distance is less than the second input distance;
perform clustering based on the locality-preserving hashes; and
for each particular cluster:
set a cluster center data block for the particular cluster; and
for each particular data block of the particular cluster, store the particular data block as a cluster member data block including a reference to the cluster center data block for the particular cluster, a list of differences of the particular data block from the cluster member data block, the cryptographic hash for the particular data block to identify the particular data block, and the locality-preserving hash for the particular data block for manipulation of the particular data block.
2. The information handling system of claim 1 , wherein the deduplication agent is further configured to, in response to finding a duplicate of a data block of a particular cluster:
store the data block as a reference to the cluster center data block for the cluster; and
store the duplicate as a reference to the data block.
3. The information handling system of claim 1 , wherein the deduplication agent is further configured to, in response to a request to receive a logical data block from the storage resource:
determine a relevant cluster center data block of a cluster for the logical data block and a stored list of differences for the cluster; and
apply the differences to the cluster center data block to return data of the logical data block.
4. The information handling system of claim 1 , wherein the deduplication agent is further configured to, in response to a request to store a new block to the storage resource:
determine if moving a cluster center for a cluster would increase storage efficiency;
determine if the new block falls within the boundaries of an existing cluster;
if the new block falls within the boundaries of an existing cluster and if moving the cluster center for such existing cluster would increase storage efficiency, move the cluster center for the existing cluster to minimize lists of differences for the existing cluster including for the new block, and recalculate the list of differences for the existing cluster;
if the new block falls outside the boundaries of the existing cluster and if moving the cluster center for another nearby cluster would increase storage efficiency, move the cluster center for the nearby cluster to include the new block, and recalculate the list of differences for the nearby cluster;
if the new block falls outside the boundaries of the existing cluster and if moving the cluster center for another nearby cluster would not increase storage efficiency, create a new one-member cluster for the new block;
if the new block falls within the boundaries of the existing cluster and if moving the cluster center for the existing cluster would not increase storage efficiency, calculate a cryptographic hash, a locality-preserving hash, and a different list for the new block; and
if the new block falls outside the boundaries of the existing cluster and if moving the cluster center for the existing cluster would not increase storage efficiency, create a new one-member cluster for the new block.
5. The information handling system of claim 1 , wherein the deduplication agent is further configured to, in response to a request to remove a block from the storage resource:
determine if moving a cluster center for the cluster including the block would increase storage efficiency;
if moving the cluster center for the cluster including the block would not increase storage efficiency, remove the block, including the cryptographic hash, the locality-preserving hash, and the difference list for the block; and
if moving the cluster center for the cluster including the block would increase storage efficiency, move the cluster center for the cluster including the block to minimize lists of differences for the cluster including the block, and recalculate the list of differences for such cluster.
6. A method comprising:
calculating a cryptographic hash for each of a plurality of data blocks stored on a storage resource;
calculating, with a locality-preserving hash function, a locality-preserving hash for each of the plurality of data blocks, wherein a distance between locality-preserving hashes of two inputs that have a first input distance between them is less than a distance between locality-preserving hashes for two inputs that have a second distance between them when the first input distance is less than the second input distance;
performing clustering based on the locality-preserving hashes; and
for each particular cluster:
setting a cluster center data block for the particular cluster; and
for each particular data block of the particular cluster, storing the particular data block as a cluster member data block including a reference to the cluster center data block for the particular cluster, a list of differences of the particular data block from the cluster member data block, the cryptographic hash for the particular data block to identify the particular data block, and the locality-preserving hash for the particular data block for manipulation of the particular data block.
7. The method of claim 6 , further comprising, in response to finding a duplicate of a data block of a particular cluster:
storing the data block as a reference to the cluster center data block for the cluster; and
storing the duplicate as a reference to the data block.
8. The method of claim 6 , further comprising, in response to a request to receive a logical data block from the storage resource:
determining a relevant cluster center data block of a cluster for the logical data block and a stored list of differences for the cluster; and
applying the differences to the cluster center data block to return data of the logical data block.
9. The method of claim 6 , further comprising, in response to a request to store a new block to the storage resource:
determining if moving a cluster center for a cluster would increase storage efficiency;
determining if the new block falls within the boundaries of an existing cluster;
if the new block falls within the boundaries of an existing cluster and if moving the cluster center for such existing cluster would increase storage efficiency, moving the cluster center for the existing cluster to minimize lists of differences for the existing cluster including for the new block, and recalculating the list of differences for the existing cluster;
if the new block falls outside the boundaries of the existing cluster and if moving the cluster center for another nearby cluster would increase storage efficiency, moving the cluster center for the nearby cluster to include the new block, and recalculating the list of differences for the nearby cluster;
if the new block falls outside the boundaries of the existing cluster and if moving the cluster center for another nearby cluster would not increase storage efficiency, creating a new one-member cluster for the new block;
if the new block falls within the boundaries of the existing cluster and if moving the cluster center for the existing cluster would not increase storage efficiency, calculating a cryptographic hash, a locality-preserving hash, and a different list for the new block; and
if the new block falls outside the boundaries of the existing cluster and if moving the cluster center for the existing cluster would not increase storage efficiency, creating a new one-member cluster for the new block.
10. The method of claim 6 , further comprising, in response to a request to remove a block from the storage resource:
determining if moving a cluster center for the cluster including the block would increase storage efficiency;
if moving the cluster center for the cluster including the block would not increase storage efficiency, removing the block, including the cryptographic hash, the locality-preserving hash, and the difference list for the block; and
if moving the cluster center for the cluster including the block would increase storage efficiency, moving the cluster center for the cluster including the block to minimize lists of differences for the cluster including the block, and recalculating the list of differences for such cluster.
11. An article of manufacture comprising:
a non-transitory computer-readable medium; and
computer-executable instructions carried on the computer-readable medium, the instructions readable by a processor, the instructions, when read and executed, for causing the processor to:
calculate a cryptographic hash for each of a plurality of data blocks stored on a storage resource;
calculate, with a locality-preserving hash function, a locality-preserving hash for each of the plurality of data blocks wherein a distance between locality-preserving hashes of two inputs that have a first input distance between them is less than a distance between locality-preserving hashes for two inputs that have a second distance between them when the first input distance is less than the second input distance;
perform clustering based on the locality-preserving hashes; and
for each particular cluster:
set a cluster center data block for the particular cluster; and
for each particular data block of the particular cluster, store the particular data block as a cluster member data block including a reference to the cluster center data block for the particular cluster, a list of differences of the particular data block from the cluster member data block, the cryptographic hash for the particular data block to identify the particular data block, and the locality-preserving hash for the particular data block for manipulation of the particular data block.
12. The article of claim 11 , the instructions for further causing the processor to, in response to finding a duplicate of a data block of a particular cluster:
store the data block as a reference to the cluster center data block for the cluster; and
store the duplicate as a reference to the data block.
13. The article of claim 11 , the instructions for further causing the processor to, in response to a request for receive a logical data block from the storage resource:
determine a relevant cluster center data block of a cluster for the logical data block and a stored list of differences for the cluster; and
apply the differences to the cluster center data block to return data of the logical data block.
14. The article of claim 11 , the instructions for further causing the processor to, in response to a request to store a new block to the storage resource:
determine if moving a cluster center for a cluster would increase storage efficiency;
determine if the new block falls within the boundaries of an existing cluster;
if the new block falls within the boundaries of an existing cluster and if moving the cluster center for such existing cluster would increase storage efficiency, move the cluster center for the existing cluster to minimize lists of differences for the existing cluster including for the new block, and recalculate the list of differences for the existing cluster;
if the new block falls outside the boundaries of the existing cluster and if moving the cluster center for another nearby cluster would increase storage efficiency, move the cluster center for the nearby cluster to include the new block, and recalculate the list of differences for the nearby cluster;
if the new block falls outside the boundaries of the existing cluster and if moving the cluster center for another nearby cluster would not increase storage efficiency, create a new one-member cluster for the new block;
if the new block falls within the boundaries of the existing cluster and if moving the cluster center for the existing cluster would not increase storage efficiency, calculate a cryptographic hash, a locality-preserving hash, and a different list for the new block; and
if the new block falls outside the boundaries of the existing cluster and if moving the cluster center for the existing cluster would not increase storage efficiency, create a new one-member cluster for the new block.
15. The article of claim 11 , the instructions for further causing the processor to, in response to a request to remove a block from the storage resource:
determine if moving a cluster center for the cluster including the block would increase storage efficiency;
if moving the cluster center for the cluster including the block would not increase storage efficiency, remove the block, including the cryptographic hash, the locality-preserving hash, and the difference list for the block; and
if moving the cluster center for the cluster including the block would increase storage efficiency, move the cluster center for the cluster including the block to minimize lists of differences for the cluster including the block, and recalculate the list of differences for such cluster.