Storing de-duplicated data with minimal reference counts
Systems and methods for adding backups to de-duplicated storage and for removing backups from de-duplicated storage are disclosed. Backups can be added to the de-duplicated storage with minimal reference count updates. Backup data sets can be removed without garbage collection processes being performed on the de-duplicated storage.
1. A method for removing backup data of a client from a backup operation, wherein backups of the data are stored in a data set, the method comprising:
selecting a backup data set for removal, wherein the backup data set has been divided into a plurality of chunks of data and wherein a fingerprint has been determined for each of the plurality of chunks;
determining if the each chuck is a candidate chunk for removal by comparing the fingerprint of each chunk against one or more manifests of previous backup data sets and one or more manifests of subsequent backup data sets, wherein if the fingerprint for the each chunk is not found within the one or more manifests of previous backup data sets and one or more manifests of subsequent backup data sets, the chunk is a candidate chunk;
decrementing a reference count associated with the candidate chunk in a reference database that stores reference counts of chunks stored in the de-duplicated storage, wherein the reference database associates reference counts with fingerprints of chunks; and
removing the candidate chunk from the de-duplicated storage if the reference count is zero after being decremented.
2. The method of claim 1 , wherein the backup data set selected for removal is selected according to a schedule.
3. The method of claim 1 , wherein the chunks of the backup data set selected for removal can be identified from a manifest of the selected back up.
4. The method of claim 1 , wherein for each chunk, referencing and comparing the fingerprint of each chunk to one or more manifests of previous backup data sets, to determine if the chunk is found within the one or more manifests of previous backup data sets.
5. The method of claim 1 , wherein for each chunk, referencing and comparing the fingerprint of each chunk to one or more manifests of subsequent backup data sets, to determine if the chunk is found within the one or more manifests of subsequent backup data sets.
6. The method of claim 1 , wherein if a chunk is not identified as a candidate chunk, no further action need be taken with respect to that chunk.
7. The method of claim 1 , wherein if the reference count of a candidate chunk is not zero after being decremented, no further action need be taken with respect to that chunk.
8. The method of claim 1 , wherein the method refrains from garbage collection after the removal of a selected backup data set to remove unreferenced chunks.
9. The method of claim 1 , wherein the previous backup data sets, the selected backup data set, and the subsequent backup data sets have been de-duplicated.
10. The method of claim 1 , wherein the fingerprint of a candidate chuck is removed if the reference count associated with the candidate chuck is decremented to zero.
11. A method for removing backup data set in a de-duplicated storage, wherein the de-duplicated storage stores previous backups, the method comprising:
selecting a backup data set for removal, wherein the backup data set has been divided into a plurality of chunks of data and wherein a fingerprint has been determined for each of the plurality of chunks;
for each chunk of data to be removed:
referencing and comparing the fingerprint of each chunk to one or more manifests of previous backup data sets and one or more manifests of subsequent backup data sets, to determine if the chunk is found within the one or more manifests of previous backup data sets and one or more manifests of subsequent backup data sets,
labeling the chunk as a candidate chunk if it is not found within the one or more manifests of previous backup data sets and one or more manifests of subsequent backup data sets;
decrementing a reference count associated with the candidate chunk in a reference database that stores reference counts of chunks stored in the de-duplicated storage when the chunk is not present in the one or more previous backups and is present in the de-duplicated storage, wherein the reference database associates reference counts with fingerprints of chunks; and
removing the candidate chunk and the associated fingerprint from the de-duplicated storage if the reference count is zero after being decremented.
12. The method of claim 11 , wherein the backup data set selected for removal is selected according to a schedule.
13. The method of claim 11 , wherein the chunks of the backup data set selected for removal can be identified from a manifest of the selected back up.
14. The method of claim 11 , wherein if a chunk is not identified as a candidate chunk, no further action need be taken with respect to that chunk.
15. The method of claim 11 , wherein if the reference count of a candidate chunk is not zero after being decremented, no further action need be taken with respect to that chunk.
16. The method of claim 11 , wherein the method refrains from garbage collection after the removal of a selected backup data set to remove unreferenced chunks.
17. The method of claim 11 , wherein the previous backup data sets, the selected backup data set, and the subsequent backup data sets have been de-duplicated.
18. The method of claim 11 , wherein the fingerprint of a candidate chuck is removed if the reference count associated with the candidate chuck is decremented to zero.