Dynamic duplication estimation for garbage collection
Embodiments are directed to perfect physical garbage collection (PPGC) process that dynamically estimates duplicate containers using a Bloom filter-based dead vector by scanning an index containing a mapping of fingerprints to a container ID for a plurality of containers; returning, for each fingerprint, a fingerprint sequence associating each fingerprint with a respective unique container ID, wherein a last entry of the sequence is preserved and the remaining entries are considered duplicates; and maintaining a duplicate array of counts of the duplicates indexed by container IDs, and wherein the duplicate array comprises a duplicate counter that keeps track of a number of live duplicated segments for each container, and further wherein a live segment is a live duplicate segment if a segment with a same fingerprint exists in another container with a higher container ID.
1. A computer-implemented method of dynamically estimating duplicate containers in a garbage collection process for use in a deduplication backup system, comprising:
scanning, in an analysis phase of the garbage collection process, an index containing a mapping of fingerprints to a sequentially numbered container ID (CID) for a plurality of containers;
returning, for each fingerprint, a fingerprint sequence associating each fingerprint with a respective unique container ID, wherein a last entry of the sequence is preserved and the remaining entries are considered duplicates; and
maintaining a duplicate array of container IDs indexed by an incremental count, wherein the incremental count is incremented for each of the container IDs, and wherein the duplicate array comprises a duplicate counter that keeps track of a number of live duplicated segments for each container, and further wherein a live segment is a live duplicate segment if a segment with a same fingerprint exists in another container with a higher container ID number relative to other container IDs.
2. The method of claim 1 further comprising tracking, in a select phase of the garbage collection process, dead segments in a dead vector by scanning in reverse numerical order the containers of the plurality of containers from the highest numbered CID to the lowest numbered CID.
3. The method of claim 2 further comprising:
determining if a segment in a container is a dead segment;
checking the dead vector to see if the fingerprint is present, and if so, incrementing the count of dead segments in CID by one; and
if the dead fingerprint is not present in the dead vector, then adding it to the dead vector.
4. The method of claim 3 wherein the dead vector is implemented using a Bloom filter.
5. The method of claim 4 further comprising determining if there is a collision in the dead vector, and if not taking the dead segment count to be the exact number of dead duplicate segments.
6. The method of claim 5 further comprising using a probability of collision of the Bloom filter to produce a dead duplicate estimate based on a uniform distribution function that is a function of the duplicate array multiplied by a percentage of dead segments.
7. The method of claim 6 wherein an estimate of live duplicates is determined by subtracting a number of dead duplicate segments from a total number of duplicate segments.
8. The method of claim 7 wherein a liveness of a container is estimated by subtracting an array of live duplicates from a total number of live segments.
9. The method of claim 8 wherein the estimate of the liveness of a container is used in a perfect physical garbage collection (PPGC) process using a perfect hash live vector with a live vector comprising a perfect hash vector, and no live instance vector.
10. The method of claim 9 wherein the liveness of the container is used by the PPGC during traversal of the plurality of containers in the select phase to select containers that meet a defined liveness threshold prior to copying forward, in a copy phase, live segments out of the selected containers.
11. A system implementing perfect physical garbage collection (PPGC) within a deduplication backup network by dynamically estimating duplicate containers in a garbage collection process for use in a deduplication backup system, comprising:
a first component scanning, in an analysis phase of the garbage collection process, an index containing a mapping of fingerprints to a sequentially numbered container ID (CID) for a plurality of containers, and returning, for each fingerprint, a fingerprint sequence associating each fingerprint with a respective unique container ID, wherein a last entry of the sequence is preserved and the remaining entries are considered duplicates; and
a second component maintaining a duplicate array of container IDs indexed by an incremental count, wherein the incremental count is incremented for each of the container IDs, and wherein the duplicate array comprises a duplicate counter that keeps track of a number of live duplicated segments for each container, and further wherein a live segment is a live duplicate segment if a segment with a same fingerprint exists in another container with a higher container ID number relative to other container IDs.
12. The system of claim 11 further comprising tracking component tracking, in a select phase of the garbage collection process, dead segments in a dead vector by scanning in reverse numerical order the containers of the plurality of containers from the highest numbered CID to the lowest numbered CID.
13. The system of claim 12 wherein the dead vector is implemented using a Bloom filter, and further comprising:
a third component determining if a segment in a container is dead, and if so incrementing a dead segment count for the dead vector by one, and if not inserting the corresponding fingerprint in the dead vector and incrementing a CID count by one.
14. The system of claim 13 wherein the third component further determines if there is a collision in the dead vector, and if not taking the dead segment count to be the exact number of dead duplicate segments, and using a probability of collision of the Bloom filter to produce a dead duplicate estimate based on a uniform distribution function that is a function of the duplicate array multiplied by a percentage of dead segments.
15. The system of claim 14 wherein an estimate of live duplicates is determined by subtracting a number of dead duplicate segments from a total number of duplicate segments, and further wherein a liveness of a container is estimated by subtracting an array of live duplicates from a total number of live segments.
16. The system of claim 15 wherein the estimate of the liveness of a container is used in a perfect physical garbage collection (PPGC) process using a perfect hash live vector with a live vector comprising a perfect hash vector, and no live instance vector.
17. The system of claim 16 wherein the liveness of the container is used by the PPGC during traversal of the plurality of containers in the select phase to select containers that meet a defined liveness threshold prior to copying forward, in a copy phase, live segments out of the selected containers.
18. A computer program product, comprising a non-transitory computer-readable medium having a computer-readable program code embodied therein, the computer-readable program code adapted to be executed by one or more processors to implement a method of performing
perfect physical garbage collection for use in a deduplication backup system that dynamically estimates duplicate containers, by:
scanning, in an analysis phase of the garbage collection process, an index containing a mapping of fingerprints to a sequentially numbered container ID (CID) for a plurality of containers;
returning, for each fingerprint, a fingerprint sequence associating each fingerprint with a respective unique container ID, wherein a last entry of the sequence is preserved and the remaining entries are considered duplicates; and
maintaining a duplicate array of container IDs indexed by an incremental count, wherein the incremental count is incremented for each of the container IDs, and wherein the duplicate array comprises a duplicate counter that keeps track of a number of live duplicated segments for each container, and further wherein a live segment is a live duplicate segment if a segment with a same fingerprint exists in another container with a higher container ID number relative to other CIDs.
19. The computer program product of claim 18 , wherein the method further comprises tracking, in a select phase of the garbage collection process, dead segments in a dead vector by scanning in reverse numerical order the containers of the plurality of containers from the highest numbered CID to the lowest numbered CID.
20. The computer program product of claim 19 , wherein the method further comprises:
determining if a segment in a container is a dead segment;
checking the dead vector to see if the fingerprint is present, and if so, incrementing the count of dead segments in CID by one; and
if the dead fingerprint is not present in the dead vector, then adding it to the dead vector.