Scalable, persistent, high performance and crash resilient metadata microservice
In a deduplicated microservices-based storage system and method running in a cloud includes frontend and backend microservice nodes that process requests from a plurality of clients and interface with a master node and one or more worker nodes connected as a network for deduplicating, storing and retrieving data from object storage, a worker node receives queries and fingerprints for data segments of a data similarity group to be deduplicated, obtains and returns the similarity group from its cache or from object storage using a key. The system compares the received fingerprints to stored fingerprints to determine non-duplicates for deduplication. The worker node updates similarity groups in its cache and in a write-ahead log with a reference to a compression region of non-duplicate segments, and periodically writes modified objects from its cache to object storage.
1. A method of handling data objects in a microservices-based deduplication system having a master node and one or more worker nodes connected as a network, each worker node having an assigned partition range of values that identify data similarity groups for which said each worker node is responsible, the method comprising:
receiving by a worker node a query related to an identified similarity group;
obtaining the similarity group from a cache of the worker node or, upon determining that the similarity group is not in the cache, obtaining the similarity group from object storage of said system; and
returning the obtained similarity group; and
wherein upon the query including fingerprints for segments of said identified similarity group and comprising a request to deduplicate the segments against previously stored segments, said method further comprises:
comparing the fingerprints with the fingerprints of previously stored segments to determine whether the received fingerprints are non-duplicates; and
updating the similarity group in the cache with a reference to a compression region of non-duplicate segments.
2. The method of claim 1 , wherein said similarity group has a one or more subgroups created by updates to the similarity group, each subgroup having an increasing subgroup identity value that identifies the order in which the subgroup was created, and wherein said obtaining said similarity group comprises obtaining the subgroup having the latest update.
3. The method of claim 2 further comprising creating a subgroup of the similarity group upon a size of the similarity group increasing beyond a preselected threshold.
4. The method of claim 3 , wherein said similarity group has a transaction identifier that is incremented each time the similarity group is updated, and said obtaining comprises obtaining the subgroup having the highest transaction identifier.
5. The method of claim 1 further comprising storing said compression region in said object storage, and said updating comprises updating the similarity group and a subgroup to reference said compression region.
6. The method of claim 1 further comprising writing incremental updates to similarity groups to a write-ahead log of said responsible worker node.
7. Then method of claim 6 further comprising periodically writing modified objects of said similarity group from said cache to object storage, and deleting said write-ahead log.
8. The method of claim 7 , wherein said worker node has a second write-ahead log, and said writing incremental updates comprises writing updates to said secondary log during said flushing.
9. The method of claim 6 further comprising, upon said worker node crashing and restarting, communicating by the worker node with the master node to obtain the worker node's assigned partition range of values and write-ahead log identifier, and applying updates from said write-ahead log to bring assigned similarity groups up to date.
10. The method of claim 9 further comprising checking the version of similarity groups in object storage, and skipping said applying updates for similarity groups from said write-ahead log that have lower version identifiers.
11. The method of claim 1 further comprising adjusting said partition range of values assigned to said worker node by reassigning a portion of said partition range of values to another worker node, and flushing and evicting from said first-mentioned worker node similarity groups having values within said reassigned portion.
12. A computer program product comprising non-transitory computer readable storage medium embodying instructions for controlling a processor perform a method of deduplicating data in a microservices deduplication system having a master node and one or more worker nodes connected as a network, each worker node having an assigned partition range of values that identify data similarity groups for which said each worker node is responsible, the method comprising:
receiving by a worker node a query related to an identified similarity group;
obtaining the similarity group from a cache of the worker node or, upon determining that the similarity group is not in the cache, obtaining the similarity group from object storage of said deduplication system; and
returning the obtained similarity group; and
wherein upon the query including fingerprints for segments of said identified similarity group and comprising a request to deduplicate the segments against previously stored segments, said method further comprises:
comparing the fingerprints with the fingerprints of previously stored segments to determine whether the received fingerprints are non-duplicates; and
updating the similarity group in the cache with a reference to a compression region of non-duplicate segments.
13. The computer program product of claim 12 , wherein said similarity group has a one or more subgroups created by updates to the similarity group, each subgroup having an increasing subgroup identity value that identifies the order in which the subgroup was created, and wherein said obtaining said similarity group comprises obtaining the subgroup having the latest update.
14. The computer program product of claim 13 further comprising creating a subgroup of a similarity group upon a size of the similarity group increasing beyond a preselected threshold.
15. The computer program product of claim 14 , wherein said similarity group has a transaction identifier that is incremented each time the similarity group is updated, and said obtaining comprises obtaining the subgroup having the highest transaction identifier.
16. The computer program product of claim 13 further comprising storing said new compression region in said object storage, and said updating comprises updating the similarity group and a subgroup to reference said compression region.
17. The computer program product of claim 13 further comprising further comprising periodically writing modified objects of said similarity group from said cache to object storage, and deleting said write-ahead log.
18. The computer program product of claim 17 , wherein said worker node has a second write-ahead log, and said writing incremental updates comprises writing updates to said secondary log during said flushing.
19. The computer program product of claim 17 further comprising, upon said worker node crashing and restarting, communicating by the worker node with the master node to obtain the worker node's assigned partition range of values and write-ahead log identifier, and applying updates from said write-ahead log to bring assigned similarity groups up to date.
20. The computer program product of claim 13 further comprising adjusting said partition range of values assigned to said worker node by reassigning a portion of said partition range of values to another worker node, and flushing and evicting from said first-mentioned worker node similarity groups having values within said reassigned portion.