Distributed garbage collection for the dedupe storage network
In the dedupe storage network priority to data replication needs to be given over the garbage collection (GC) activity. For this purpose GC running on any site should not impose any locking on its peer sites for replication. For replication activity GC activity must be completely transparent. In the dedupe storage network, the retention policy of a replicated image is controlled by the site where the image was originated. Distributed GC running on the originating site can only inform the replication sites the list of expired replicated images and cleanup of replicated garbage chunks from its remote FS database for corresponding replication sites. But it cannot cleanup garbage chunks from replication sites. Garbage chunks can only be cleaned by the local GC running on that site.
1. A computer-implemented method of a dedupe file system comprising:
replicating a dedupe file system specific metadata of a backup image;
replicating one or more data chunks of a backup image;
replicating the backup application specific metadata;
advertising to the backup application that the backup image is read ready,
downloading the dedupe file system specific metadata of the backup image; and
wherein when a when a replicated image is expired:
cleaning up a local view of a replicated file system, with an onsite-only garbage collection operation, by cleaning up a remote file system database;
informing a remote site about an expired image; and
cleaning up the replicated image from a local file system of the remote site while the onsite-only garbage collection operation is implemented.
2. A computer-implemented method of handling garbage data chunks in a replication operation of a dedupe file system comprising: determining when a garbage collection operation is initiated; recording the initiation time of the garbage collection
when the garbage collection operation is in a data deletion state, for each garbage data chunk implementing the following steps:
acquiring, with the garbage collection operation, a write lock for the data chunk;
determining that a hardlink count of data chunk;
if the hardlink count is one (1), marking the data chunk as garbage data chunk; and
moving, with the garbage collection operation, the garbage data chunk to a temporary trash directory;
deleting, with the garbage collection operation, the dedupe file system specific metadata of backup images, which are expired;
listing, with the garbage collection operation, one or more new backup images created as a result of a replication operation after the initiation time of garbage collection operation, and wherein the listing, with the garbage collection operation, of one or more new backup images created as a result of a replication operation is implemented at the end of a data deletion state, wherein the replication operation comprises an upload operation, a download operation or a backup operation;
for each such new backup image, reclaiming, with the garbage collection operation, one or more data chunks included in a backup image that is a part of the temporary trash directory, and wherein the garbage collection operation recovers the one or more backup data chunks to the dedupe file system; and
deleting, with the garbage collection operation, all the remaining data chunks present in the temporary trash directory.
3. A computer-implemented method of handling garbage data chunks in a replication operation of a dedupe file system comprising:
determining when a garbage collection operation is initiated;
recording the initiation time of the garbage collection
when the garbage collection operation is in a data deletion state, for each garbage data chunk implementing the following steps:
acquiring, with the garbage collection operation, a write lock for the data chunk;
determining that a hardlink count of data chunk;
if the hardlink count is one (1), marking the data chunk as garbage data chunk; and
moving, with the garbage collection operation, the garbage data chunk to a temporary trash directory;
deleting, with the garbage collection operation, the dedupe file system specific metadata of backup images, which are expired;
listing, with the garbage collection operation, one or more new backup images created as a result of a replication operation after the initiation time of garbage collection operation;
for each such new backup image, reclaiming, with the garbage collection operation, one or more data chunks included in a backup image that is a part of the temporary trash directory, and wherein the garbage collection operation recovers the one or more backup data chunks to the dedupe file system; and
deleting, with the garbage collection operation, all the remaining data chunks present in the temporary trash directory.