System and method for asynchronous cleaning of data objects on cloud partition in a file system with deduplication
A data management device includes a persistent storage and a processor. The persistent storage includes meta-data of data stored in a long term retention (LTR) storage. The processor obtains a file storage request for a file and deduplicates the file against segments stored in the LTR storage while performing garbage collection on the LTR storage. Performing garbage collection includes deleting segments of the data stored in the LTR storage using the meta-data. The meta-data is not stored in the LTR storage.
1. A data management device, comprising:
a persistent storage comprising meta-data of data stored in a long term retention (LTR) storage; and
a processor programmed to:
obtain a file storage request for a file; and
deduplicate the file against segments stored in the LTR storage while performing garbage collection on the LTR storage,
wherein deduplicating the file against segments stored in the LTR storage while performing garbage collection on the LTR storage comprises:
identifying data objects stored in the LTR storage for deletion,
identifying meta-data objects that store meta-data that is associated with the identified data objects,
copying forward the meta-data in the identified meta-data objects that is not associated with the identified data objects to obtain updated meta-data objects,
deduplicating the file using the updated meta-data objects while the identified data objects are stored in the LTR storage, and
after deduplicating the file, deleting the identified data objects from the LTR storage, wherein deleting the identified data objects from the LTR storage comprises:
generating a deletion object based on the identified data objects; and
storing the deletion object in the LTR storage, wherein the deletion object comprises a predetermined byte sequence distinguishing the deletion object from the data objects in the LTR storage,
wherein the meta-data is not stored in the LTR storage,
wherein data is stored in a deduplicated object storage comprising a plurality of data objects comprising the data, wherein each data object of the plurality of data objects comprises a plurality of segments of files of the data, and
wherein at least one segment of the segments of the files is used to reconstruct a first file of the files of the data and a second file of the files of the data.
2. The data management device of claim 1 , wherein the meta-data is stored in meta-data objects stored in an active storage of the persistent storage.
3. The data management device of claim 2 , wherein the meta-data comprises:
fingerprints of segments of files stored in the LTR storage; and
data object identification information that enables data objects that store the segments to be retrieved from the LTR storage.
4. The data management device of claim 1 , wherein the LTR storage is a cloud storage.
5. The data management device of claim 1 , wherein performing garbage collection on the LTR storage comprises:
deleting a plurality of data objects stored in the LTR storage based on object identification information stored in the deletion object.
6. The data management device of claim 5 , wherein the object identification information comprises:
a first entry comprising:
a logical container identifier; and
an offset that specifies a location of a data region within a logical container identified by the logical container identifier.
7. The data management device of claim 5 , wherein performing garbage collection on the LTR storage further comprises:
iterating through objects stored in the LTR storage starting at a last processed deletion object in order of increasing identifier value, and
identifying the deletion object based on the iterating.
8. The data management device of claim 5 , wherein performing garbage collection on the LTR storage further comprises:
after deleting the plurality of data objects, updating a last processed deletion object identifier stored in an active storage based on an identifier of the deletion object,
wherein the meta-data is stored in the active storage,
wherein the active storage and the LTR storage are different storages.
9. The data management device of claim 5 , wherein performing garbage collection on the LTR storage further comprises:
after deleting the plurality of data objects, marking the deletion object for deletion.
10. The data management device of claim 9 , wherein marking the deletion object for deletion comprises:
iterating through objects stored in the LTR storage, starting at an object having a smallest identifier value, in order of numerically increasing identifier value until an object having an identifier value specified by a last processed deletion object identifier is reached, and
mark each object of the iterated objects that are deletion objects for deletion.
11. The data management device of claim 1 , wherein the identified data object for deletion is a portion of a second file, wherein the second file is different from the file.
12. A method of operating a data management device, comprising:
obtaining, by the data management device, a file storage request specifying a file; and
deduplicating, by the data management device, the file against segments stored in a long term retention (LTR) storage while performing garbage collection on the LTR storage,
wherein deduplicating the file against segments stored in the LTR storage while performing garbage collection on the LTR storage comprises:
identifying, by a data management device, data objects stored in the LTR storage for deletion,
identifying, by a data management device, meta-data objects that store meta-data that is associated with the identified data objects,
copying forward, by a data management device, the meta-data in the identified meta-data objects that is not associated with the identified data objects to obtain updated meta-data objects,
deduplicating, by a data management device, the file using the updated meta-data objects while the identified data objects are stored in the LTR storage, and
after deduplicating the file, deleting, by a data management device, the identified data objects from the LTR storage, wherein deleting the identified data objects from the LTR storage comprises:
generating a deletion object based on the identified data objects; and
storing the deletion object in the LTR storage, wherein the deletion object comprises a predetermined byte sequence distinguishing the deletion object from the data objects in the LTR storage,
wherein the meta-data is not stored in the LTR storage,
wherein data is stored in a deduplicated object storage comprising a plurality of data objects comprising the data, wherein each data object of the plurality of data objects comprises a plurality of segments of files of the data, and
wherein at least one segment of the segments of the files is used to reconstruct a first file of the files of the data and a second file of the files of the data.
13. A non-transitory computer readable medium comprising computer readable program code, which when executed by a computer processor enables the computer processor to perform a method for operating a data management device, the method comprising:
obtaining, by the data management device, a file storage request for a file; and
deduplicating, by the data management device, the file against segments stored in a long term retention (LTR) storage while performing garbage collection on the LTR storage,
wherein deduplicating the file against segments stored in the LTR storage while performing garbage collection on the LTR storage comprises:
identifying, by a data management device, data objects stored in the LTR storage for deletion,
identifying, by a data management device, meta-data objects that store meta-data that is associated with the identified data objects,
copying forward, by a data management device, the meta-data in the identified meta-data objects that is not associated with the identified data objects to obtain updated meta-data objects,
deduplicating, by a data management device, the file using the updated meta-data objects while the identified data objects are stored in the LTR storage, and
after deduplicating the file, deleting, by a data management device, the identified data objects from the LTR storage, wherein deleting the identified data objects from the LTR storage comprises:
generating a deletion object based on the identified data objects; and
storing the deletion object in the LTR storage, wherein the deletion object comprises a predetermined byte sequence distinguishing the deletion object from the data objects in the LTR storage,
wherein the meta-data is not stored in the LTR storage,
wherein data is stored in a deduplicated object storage comprising a plurality of data objects comprising the data, wherein each data object of the plurality of data objects comprises a plurality of segments of files of the data, and
wherein at least one segment of the segments of the files is used to reconstruct a first file of the files of the data and a second file of the files of the data.