Methods and systems of a dedupe file-system garbage collection
In one embodiment, a computer-implemented method of implementing a dedupe file system with constant ingestion and retrieval times for objects in dedupe file system achieved by synchronizing a garbage collection (GC) thread and reader (restore), writer (backup) threads in a dedupe file system includes generating, with at least one processor, a state machine for dedupe file system that coordinates concurrent data ingestion due to writers and data deletion due to garbage collector. The state machine has three states. “Dormant” state when the GC thread is inactive and all writers freely ingest data into dedupe file system without any need for synchronization with GC. “Data gathering” state when the GC thread determines the dedupe chunks for deletion in a garbage list and writers check for data chunks in the garbage list and filters out processed data.
1. A computerized system of concurrently synchronizing a garbage collection thread and a writer threads in a dedupe file system in a data gathering state comprising:
a processor configured to execute instructions;
a memory containing instructions when executed on the processor, causes the processor to perform operations that:
while the dedupe file system is in a data gathering state,
a garbage collector thread concurrently working with writer threads, generate, with at least one processor, a garbage list of data chunks that are candidates for deletion by a garbage collector thread, wherein the garbage collector thread enumerates all backups on the data store, and wherein the garbage collector thread traverses a list of valid backups and removes any data chunks of the list of valid backups from an eraser database of the dedupe file system; and
with the writer threads, referring to the garbage list of data chunks while ingesting data by:
matching the data chunks with those present in the garbage list;
filtering out the matched data chunks from garbage list of data chunks;
setting, with the garbage collector thread, the dedupe file system to a data deletion state; and
setting the writer threads to ingest data into the dedupe file system in synchronization with garbage collector thread.
2. A computerized system of synchronizing a garbage collection thread and a writer threads in a dedupe file system in Data deletion state comprising:
a processor configured to execute instructions;
a memory containing instructions when executed on the processor, causes the processor to perform operations that:
provide one or more writer threads concurrently working with garbage collector thread, referring to the garbage list of data chunks while ingesting data by:
matching the data chunks with those present in the garbage list;
for the matched data chunk add one or more hard links to the data chunk file in a temporary location, and wherein the hard links lock the data chunk file from deletion, wherein the hard links are a directory entry that associates a name with a data chunk file on a file system of the computerized system;
with a garbage collector thread:
iterate through the garbage list; and
obtain an exclusive access of each data chunk and delete any data chunk that is not marked by the one or more writer threads as having two hard links.