De-duplication for a global coherent de-duplication repository
View Patent ↗Example methods and apparatus associated with data de-duplication for a global coherent de-duplication repository are provided. In one example a request related to data de-duplication is transmitted to a plurality of nodes associated with the global coherent de-duplication repository. Responses to the request are received from at least a subset of nodes in the plurality of nodes. Affinity scores are assigned to nodes of the subset of nodes based, at least in part, on affinity data from the responses. A node is selected to perform the request related to de-duplication from the subset of nodes of the plurality of nodes based, at least in part, on the affinity score assigned to the nodes.
1. A non-transitory computer-readable medium storing computer-executable instructions that when executed by a data de-duplication apparatus control the data de-duplication apparatus to perform a method, the method comprising:
transmitting a request associated with de-duplication to a plurality of nodes associated with a global coherent de-duplication repository, where the plurality of nodes are arranged in groups associated with the global coherent de-duplication repository;
receiving responses to the request from at least a subset of nodes of the plurality of nodes associated with the global coherent de-duplication repository;
calculating affinity scores for nodes of the subset of nodes based, at least in part, on affinity factors received in the responses from the nodes, where calculating an affinity score employs a group calculation that is used when a node belongs to a group;
calculating a total cost of de-duplication for a node by determining a number of unstored blocks by subtracting sub-blocks to be deduplicated by a number of previously stored blocks, and multiplying the number of unstored blocks by a cost per sub-block storage;
including the total cost of de-duplication for the node in the affinity score for the node; and
selecting a node from the subset of the plurality of nodes to perform the request based, at least in part, on the affinity scores calculated for the nodes.
2. The computer-readable medium of claim 1 , where an affinity score is based, at least in part, on a plurality of affinity factors, and where the affinity factors are weighted.
3. The computer-readable medium of claim 2 , where a first affinity factor is weighted with a first weight and a second affinity factor is weighted with a second weight, and where the first weight is different than the second weight.
4. The computer-readable medium of claim 1 , where a first node is selected to perform a first portion of the request and a second node is selected to perform a second portion of the request.
5. A data de-duplication apparatus, comprising:
a processor;
a memory;
a set of logics;
an interface configured to connect the processor, the memory, and the set of logics,
the set of logics comprising:
a communication logic configured to communicate a de-duplication request to a plurality of nodes associated with a global coherent de-duplication repository;
an affinity logic configured to calculate affinity scores for nodes in the plurality of nodes based, at least in part, on affinity data, where the affinity data includes affinity factors received from one or more of the plurality of nodes;
a selection logic configured to select at least one node to perform de-duplication;
a grouping logic configured to arrange the plurality of nodes in groups, where the affinity score for a node is affected by a group the node belongs to, by communicating with the grouping logic; and
where the affinity logic is configured to calculate a total cost of de-duplication for a node by determining a number of unstored blocks by subtracting sub-blocks to be deduplicated by a number of previously stored blocks, and multiplying the number of unstored blocks by a cost per sub-block storage, where the total cost of de-duplication for the node is included in the affinity score for the one or more nodes.
6. The apparatus of claim 5 , where affinity data is based, at least in part, on a plurality of affinity factors, where the affinity factors are weighted.
7. A method, comprising:
transmitting a request associated with de-duplication to a plurality of nodes associated with a global coherent de-duplication repository;
receiving responses to the request from at least a subset of nodes of the plurality of nodes associated with the global coherent de-duplication repository;
identifying at least one node of the subset of nodes that belongs to a group;
determining if the group associates the at least one node with an affinity score group calculation;
calculating affinity scores for nodes of the subset of nodes based, at least in part, on at least one affinity factor received in the responses from the nodes, where the at least one node's affinity score is based, at least in part, on the affinity score group calculation;
calculating a total cost of de-duplication for a node by determining a number of unstored blocks by subtracting sub-blocks to be deduplicated by a number of previously stored blocks, and multiplying the number of unstored blocks by a cost per sub-block storage;
including the total cost of de-duplication for the node in the affinity score for the node; and
selecting a node from the subset of the plurality of nodes to perform the request based, at least in part, on the affinity scores calculated for the nodes.