Verification microservice for a deduplicated object storage system with worker node estimation
Described is a system for detecting corruption in a deduplicated object storage system accessible by one or more microservices while minimizing costly read operations on objects. A controller module and one or more worker nodes execute verification paths in concert with each other to identify object corruptions. The controller module estimates a number of worker nodes required for execution of different types of verification paths.
1. A computer-implemented method, comprising:
determining a first total memory size of worker node memory identified for storage of slice recipe names currently located in object storage of a deduplicated storage system, wherein each slice recipe name references a similarity group and each similarity group is pre-assigned to a worker node;
determining a second total memory size of the worker node memory identified for storage of the referenced similarity groups; and
estimating a number of worker nodes to be instantiated based on the first total memory size or the second total memory size of worker node memory for performance of a slice recipe verification path or a similarity group verification path, the first total memory size or the second total memory size of worker node memory the first total memory size of worker node memory supporting the slice recipe verification path and the second total memory size of worker node memory supporting the similarity group verification path.
2. The computer-implemented method of claim 1 , further comprising:
detecting occurrence of a worker node memory condition indicating the second total memory size of worker node memory for the referenced similarity groups exceeds the first total memory size of worker node memory for the slice recipe names.
3. The computer-implemented method of claim 2 , wherein the first total memory size of worker node memory for the slice recipe names includes memory sizes of respective memory partitions internal to a plurality of the worker nodes and wherein the second total memory size of worker node memory for the referenced similarity groups includes the memory sizes of one or more of the respective memory partitions internal to the plurality of the worker nodes.
4. The computer-implemented method of claim 2 , further comprising:
tracking a running tally of a total memory size of worker node memory identified for storage of a subset of the referenced similarity groups; and
comparing the running tally of the total memory size of worker node memory for the subset of referenced similarity groups to the first total memory size of worker node memory identified for the slice recipe names; and
detecting an occurrence of the worker node memory condition based upon the running tally for the subset of referenced similarity groups exceeding the first total memory size of worker node memory identified for the slice recipe names.
5. The computer-implemented method of claim 4 , further comprising:
instantiating one or more worker nodes with respective internal memory partitions for storing the slice recipes names that corresponds to first total memory size of worker node memory, the respective, instantiated worker nodes for support of performance of the slice recipe verification path.
6. The computer-implemented method of claim 4 , further comprising:
detecting absence of the worker node memory condition based upon a final tally of the total memory size of worker node memory for all referenced similarity groups not exceeding the first total memory size of worker node memory identified for the slice recipe names.
7. The computer-implemented method of claim 6 , further comprising:
instantiating one or more worker nodes with respective internal memory partitions for storing the referenced similarity groups that corresponds to the final tally the total memory size of worker node memory for all referenced similarity groups, the respective, instantiated worker nodes for support of performance of the similarity group verification path.
8. A system comprising:
one or more processors; and
a non-transitory computer readable medium storing a plurality of instructions, which when executed, cause the one or more processors to:
determine a first total memory size of worker node memory identified for storage of slice recipe names currently located in object storage of a deduplicated storage system, wherein each slice recipe name references a similarity group and each similarity group is pre-assigned to a worker node;
determine a second total memory size of the worker node memory identified for storage of the referenced similarity groups; and
estimate a number of worker nodes to be instantiated based on the first total memory size or the second total memory size of worker node memory for performance of a slice recipe verification path or a similarity group verification path, the first total memory size of work node memory supporting the slice recipe verification path and the second total memory size of worker node memory supporting the similarity group verification path.
9. The system of claim 8 , wherein the plurality of instructions, when executed, further cause the one or more processors to:
detect occurrence of a worker node memory condition indicating the second total memory size of worker node memory for the referenced similarity groups exceeds the first total memory size of worker node memory for the slice recipe names.
10. The system of claim 9 , wherein the first total memory size of worker node memory for the slice recipe names includes memory sizes of respective memory partitions internal to a plurality of the worker nodes and wherein the second total memory size of worker node memory for the referenced similarity groups includes the memory sizes of one or more of the respective memory partitions internal to the plurality of the worker nodes.
11. The system of claim 9 , wherein the plurality of instructions, when executed, further cause the one or more processors to:
track a running tally of a total memory size of worker node memory identified for storage of a subset of the referenced similarity groups; and
compare the running tally of the total memory size of worker node memory for the subset of referenced similarity groups to the first total memory size of worker node memory identified for the slice recipe names; and
detect an occurrence of the worker node memory condition based upon the running tally for the subset of referenced similarity groups exceeding the first total memory size of worker node memory identified for the slice recipe names.
12. The system of claim 11 , wherein the plurality of instructions, when executed, further cause the one or more processors to:
instantiate one or more worker nodes with respective internal memory partitions for storing the slice recipes names that corresponds to first total memory size of worker node memory, the respective, instantiated worker nodes for support of performance of the slice recipe verification path.
13. The system of claim 11 , wherein the plurality of instructions, when executed, further cause the one or more processors to:
detect absence of the worker node memory condition based upon a final tally of the total memory size of worker node memory for all referenced similarity groups not exceeding the first total memory size of worker node memory identified for the slice recipe names.
14. The system of claim 13 , wherein the plurality of instructions, when executed, further cause the one or more processors to:
instantiate one or more worker nodes with respective internal memory partitions for storing the referenced similarity groups that corresponds to the final tally the total memory size of worker node memory for all referenced similarity groups, the respective, instantiated worker nodes for support of performance of the similarity group verification path.
15. A computer program product comprising a non-transitory computer-readable medium having a computer-readable program code embodied therein to be executed by one or more processors, the program code including instructions to:
determine a first total memory size of worker node memory identified for storage of slice recipe names currently located in object storage of a deduplicated storage system, wherein each slice recipe name references a similarity group and each similarity group is pre-assigned to a worker node;
determine a second total memory size of the worker node memory identified for storage of the referenced similarity groups; and
estimate a number of worker nodes to be instantiated based on the first total memory size or the second total memory size of worker node memory for performance of a slice recipe verification path or a similarity group verification path, the first total memory size of worker node memory supporting the slice recipe verification path and the second total memory size of worker node memory supporting the similarity group verification path.
16. The computer program product of claim 15 , wherein the program code includes further instructions to:
detect occurrence of a worker node memory condition indicating the second total memory size of worker node memory for the referenced similarity groups exceeds the first total memory size of worker node memory for the slice recipe names.
17. The computer program product of claim 16 , wherein the first total memory size of worker node memory for the slice recipe names includes memory sizes of respective memory partitions internal to a plurality of the worker nodes and wherein the second total memory size of worker node memory for the referenced similarity groups also includes the memory sizes of one or more of the respective memory partitions internal to the plurality of the worker nodes.
18. The computer program product of claim 16 , wherein the program code includes further instructions to:
track a running tally of a total memory size of worker node memory identified for storage of a subset of the referenced similarity groups; and
compare the running tally of the total memory size of worker node memory for the subset of referenced similarity groups to the first total memory size of worker node memory identified for the slice recipe names; and
detect an occurrence of the worker node memory condition based upon the running tally for the subset of referenced similarity groups exceeding the first total memory size of worker node memory identified for the slice recipe names.
19. The computer program product of claim 18 , wherein the program code includes further instructions to:
instantiate one or more worker nodes with respective internal memory partitions for storing the slice recipes names that corresponds to first total memory size of worker node memory, the respective, instantiated worker nodes for support of performance of the slice recipe verification path.
20. The computer program product of claim 18 , wherein the program code includes further instructions to:
detect absence of the worker node memory condition based upon a final tally of the total memory size of worker node memory for all referenced similarity groups not exceeding the first total memory size of worker node memory identified for the slice recipe names; and
instantiate one or more worker nodes with respective internal memory partitions for storing the referenced similarity groups that corresponds to the final tally the total memory size of worker node memory for all referenced similarity groups, the respective, instantiated worker nodes for support of performance of the similarity group verification path.