Data fingerprint distribution on a data storage system
Fingerprints of data portions are distributed in a balanced manner across active controllers of a data storage system, and may be done so in such a manner that, when a new active controller is added to the system, fingerprint ownership and movement between pre-existing active controllers, and active controllers overall, is minimized When a new active controller is added to the system and fingerprints are redistributed, no fingerprint ownership may be re-assigned between pre-existing active controllers and no fingerprints may be moved between pre-existing active controllers, for example, between local memories of the active controller.
1. For a data storage system in which fingerprints are determined for data portions of a predetermined size, a method of performing deduplication comprising:
grouping potential fingerprints into a plurality of fingerprint groups, the plurality of fingerprint groups representing a balanced distribution of the potential fingerprints;
for each of the plurality of fingerprint groups, assigning ownership of the fingerprint groups to one of a plurality of active controllers on the data storage system such that each active controller is assigned an equal number of the plurality of storage groups;
in response to adding an active storage controller to the data storage system, re-assigning ownership of at least one fingerprint group to the added active controller such that each active controller is still assigned an equal number of the plurality of storage groups and no fingerprint groups are reassigned between storage controllers that already were active prior to the adding of the active storage controller; and
performing deduplication on the storage system based at least in part on the assignment of ownership of fingerprint groups to active controllers.
2. The method of claim 1 , wherein a maximum percentage of fingerprint groups that are re-assigned to the added active storage controllers is: 100/x, wherein x is a number of active storage controllers on the data storage system after the adding of the active controller.
3. The method of claim 1 , further comprising:
for at least each fingerprint determined for a data portion stored in the data storage system, storing the fingerprint in a fingerprint table of fingerprints in a local memory of an active storage controller determined to own the fingerprint group to which the fingerprint belongs.
4. The method of claim 3 , wherein the data storage system has a predefined potential number of active storage controllers, the method further comprising:
populating a second table including a plurality of entries, each entry corresponding to one or more of the plurality of fingerprint groups and specifying, for each of the potential number of active storage controllers, the storage controller determined to own the fingerprint group.
5. The method of claim 4 , wherein the plurality of entries of the second table do not change in response to adding another active storage controller of the potential number of active storage controllers to the data storage system.
6. The method of claim 4 , further comprising:
receiving a first data portion;
determining a first fingerprint of the data portion; and
determining, using the second table, a first active storage controller of the plurality of active storage controllers that owns the first fingerprint,
wherein performing deduplication includes using a first fingerprint table in the local memory of the first active storage controller to perform deduplication processing of the first data portion.
7. The method of claim 1 , wherein the data storage system has a predefined potential number of active storage controllers, and the method further comprising:
determining a quantity of fingerprint groups to be a lowest common multiple of each potential number of active storage controllers or a multiple thereof.
8. The method of claim 1 , wherein the fingerprints are generated for the data portions using a hash function that produces a random distribution of fingerprint values in a space of possible fingerprint values.
9. A data storage system in which fingerprints are determined for data portions of a predetermined size, the data system comprising:
a plurality of active storage controllers;
one or more processors; and
a memory comprising code stored thereon that, when executed, performs a method comprising:
grouping potential fingerprints into a plurality of fingerprint groups, the plurality of fingerprint groups representing a balanced distribution of the potential fingerprints;
for each of the plurality of fingerprint groups, assigning ownership of the fingerprint groups to one of the plurality of active controllers on the data storage system such that each active controller is assigned an equal number of the plurality of storage groups;
in response to adding an active storage controller to the data storage system, re-assigning ownership of at least one fingerprint group to the added active controller such that each active controller is still assigned an equal number of the plurality of storage groups and no fingerprint groups are reassigned between storage controllers that already were active prior to the adding of the active storage controller; and
performing deduplication on the storage system based at least in part on the assignment of ownership of fingerprint groups to active controllers.
10. The system of claim 9 , wherein a maximum percentage of fingerprint groups that are re-assigned to the added active storage controllers is: 100/x, wherein x is a number of active storage controllers on the data storage system after the adding of the active controller.
11. The system of claim 9 , wherein the method further comprises:
for at least each fingerprint determined for a data portion stored in the data storage system, storing the fingerprint in a fingerprint table of fingerprints in a local memory of an active storage controller determined to own the fingerprint group to which the fingerprint belongs.
12. The system of claim 11 , wherein the data storage system has a predefined potential number of active storage controllers, and
wherein the method further comprises populating a second table including a plurality of entries, each entry corresponding to one or more of the plurality of fingerprint groups and specifying, for each of the potential number of active storage controllers, the storage controller determined to own the fingerprint group.
13. The system of claim 12 , wherein the plurality of entries of the second table do not change in response to adding another active storage controller of the potential number of active storage controllers to the data storage system.
14. The system of claim 12 , wherein the method further comprises:
receiving a first data portion;
determining a first fingerprint of the data portion; and
determining, using the second table, a first active storage controller of the plurality of active storage controllers that owns the first fingerprint,
wherein performing deduplication includes using a first fingerprint table in the local memory of the first active storage controller to perform deduplication processing of the first data portion.
15. The system of claim 9 , wherein the data storage system has a predefined potential number of active storage controllers, and
wherein the method further comprises determining a quantity of fingerprint groups to be a lowest common multiple of each potential number of active storage controllers or a multiple thereof.
16. The system of claim 9 , wherein the fingerprints are generated for the data portions using a hash function that produces a random distribution of fingerprint values in a space of possible fingerprint values.
17. One or more non-transitory computer-readable media having software stored thereon for a data storage system on which fingerprints are determined for data portions of a predetermined size, the software comprising:
executable code that groups potential fingerprints into a plurality of fingerprint groups, the plurality of fingerprint groups representing a balanced distribution of the potential fingerprints;
executable code that, for each of the plurality of fingerprint groups, assigns ownership of the fingerprint groups to one of a plurality of active controllers on the data storage system such that each active controller is assigned an equal number of the plurality of storage groups;
executable code that, in response to adding an active storage controller to the data storage system, re-assigns ownership of at least one fingerprint group to the added active controller such that each active controller is still assigned an equal number of the plurality of storage groups and no fingerprint groups are reassigned between storage controllers that already were active prior to the adding of the active storage controller; and
executable code that performs deduplication on the storage system based at least in part on the assignment of ownership of fingerprint groups to active controllers.
18. The one or more non-transitory computer-readable media of claim 17 , wherein a maximum percentage of fingerprint groups that are re-assigned to the added active storage controllers is: 100/x, wherein x is a number of active storage controllers on the data storage system after the adding of the active controller.
19. The one or more non-transitory computer-readable media of claim 17 , the software further comprising:
executable code that, for at least each fingerprint determined for a data portion stored in the data storage system, stores the fingerprint in a fingerprint table of fingerprints in a local memory of an active storage controller determined to own the fingerprint group to which the fingerprint belongs.
20. The one or more non-transitory computer-readable media of claim 17 , wherein the data storage system has a predefined potential number of active storage controllers, and wherein the software further comprises:
executable code that determines a quantity of fingerprint groups to be a lowest common multiple of each potential number of active storage controllers or a multiple thereof.