Verification microservice for execution of a slice verification path by worker nodes
Described is a system for a verification microservice engine for generating and deploying a controller module and one or more worker nodes to detect corruption in a deduplicated object storage system accessible by one or more microservices while minimizing costly read operations on objects. A worker node(s) stores slice recipes into worker node memory that correspond to slice recipe names verified by a controller module. For each slice recipe, the worker node(s) detects whether slice recipe corruptions, similarity group corruptions and/or compression region corruptions exist.
1. A computer-implemented method, comprising:
storing, by a worker node, at least one slice recipe into worker node memory from object storage, the at least one loaded slice recipe corresponding to a slice recipe name received from a controller module;
receiving, by the worker node, one or more similarity group identifiers from the controller module;
detecting, by the worker node:
(i) whether a slice recipe corruption exists for each slice recipe in worker node memory based on a comparison between metadata of the respective slice recipe and metadata of a corresponding similarity group identified by the controller module,
(ii) whether a similarity group corruption exists based on a comparison between a first fingerprint array and a compression region name in the metadata of the corresponding similarity group, and
(iii) whether a compression region corruption exists based on a comparison between the compression region name and object storage; and
sending, by the worker node, one or more instances of slice recipe, similarity group and compression region corruptions to the controller module.
2. The computer-implemented method of claim 1 , further comprising:
receiving, by the worker node, one or more slice recipe names identified by the controller module as needing to be verified, the one or more identified slice recipe names corresponding to respective similarity groups pre-assigned to the worker node;
loading into worker node memory, by the worker node, a slice recipe from object storage that corresponds to the slice recipe name;
accessing, by the worker node, a similarity group identifier in the loaded slice recipe; and
storing, by the worker node, the loaded slice recipe in a slice recipe list associated with the similarity group identifier.
3. The computer-implemented method of claim 2 , further comprising:
sorting, by the worker node, the slice recipe, that were loaded into worker memory, into slice recipe lists associated with a respective similarity group identifiers; and
sending, by the worker node, a completion message to the controller module notifying the controller module that sorting slice recipes is complete.
4. The computer-implemented method of claim 1 , further comprising:
identifying, by the worker node, a corruption of a respective slice recipe based on detecting a fingerprint within a fingerprint array in metadata of the respective slice recipe does not match any fingerprints stored amongst a plurality of fingerprint arrays in metadata of a similarity group associated with the respective slice recipe, each fingerprint array including a fingerprint indexed to a plurality of corresponding segment sizes, the slice recipe corruption indicating that the fingerprint in the respective slice recipe that fails to match has been improperly modified.
5. The computer-implemented method of claim 1 , further comprising:
determining, by the worker node, a temporary hash score based on a particular fingerprint array from a plurality of fingerprint arrays stored in metadata of a respective similarity group, wherein each fingerprint array comprises a fingerprint indexed to a plurality of corresponding segment sizes;
accessing, by the worker node, a compression region name in the metadata of the respective similarity group, the accessed compression region name associated with the particular fingerprint array; and
identifying, by the worker node, a corruption of the respective similarity group based on the temporary hash score failing to match the accessed compression region name, the similarity group corruption indicating one or more portions of the particular fingerprint array in the metadata of the respective similarity group have been improperly modified.
6. The computer-implemented method of claim 5 , further comprising:
calling, by the worker node, object storage according to the accessed compression region name in response to determining that the temporary hash score matches the accessed compression region name; and
identifying, by the worker node, the compression region corruption on a failure to retrieve a compression region in object storage identified by the accessed compression region name.
7. The computer-implemented method of claim 1 , further comprising:
sending to the controller module, by the worker node, a message indicating completion of all attempts for identifying existence of respective slice recipe and similarity group corruption verifications.
8. A system comprising:
one or more processors; and
a non-transitory computer readable medium storing a plurality of instructions, which when executed, cause the one or more processors to:
store, by a worker node, at least one slice recipe into worker node memory from object storage, the at least one loaded slice recipe corresponding to a slice recipe name received from a controller module;
receive, by the worker node, one or more similarity group identifiers from the controller module;
detect, by the worker node:
(i) whether a slice recipe corruption exists for each slice recipe in worker node memory based on a comparison between metadata of the respective slice recipe and metadata of a corresponding similarity group identified by the controller module,
(ii) whether a similarity group corruption exists based on a comparison between a first fingerprint array and a compression region name in the metadata of the corresponding similarity group, and
(iii) whether a compression region corruption exists based on a comparison between the compression region name and object storage; and
send, by the worker node, one or more instances of slice recipe, similarity group and compression region corruptions to the controller module.
9. The system of claim 8 , wherein the plurality of instructions, when executed, further cause the one or more processors to:
receive, by the worker node, one or more slice recipe names identified by the controller module as needing to be verified, the one or more identified slice recipe names corresponding to respective similarity groups pre-assigned to the worker node;
load into worker node memory, by the worker node, a slice recipe from object storage that corresponds to the slice recipe name;
access, by the worker node, a similarity group identifier in the loaded slice recipe; and
store, by the worker node, the loaded slice recipe in a slice recipe list associated with the similarity group identifier.
10. The system of claim 9 , wherein the plurality of instructions, when executed, further cause the one or more processors to:
sort, by the worker node, the loaded slice recipe loaded into worker node memory into slice recipe lists associated with a respective similarity group identifiers; and
send, by the worker node, a completion message to the controller module notifying the controller module that sorting slice recipes is complete.
11. The system of claim 8 , wherein the plurality of instructions, when executed, further cause the one or more processors to:
identify, by the worker node, a corruption of a respective slice recipe based on detecting a fingerprint within a fingerprint array in metadata of the respective slice recipe does not match any fingerprints stored amongst a plurality of fingerprint arrays in metadata of a similarity group associated with the respective slice recipe, the fingerprint array including a fingerprint indexed to a plurality of corresponding segment sizes, the slice recipe corruption indicating that the fingerprint in the respective slice recipe that fails to match has been improperly modified.
12. The system of claim 8 , wherein the plurality of instructions, when executed, further cause the one or more processors to:
determine, by the worker node, a temporary hash score based on a particular fingerprint array from a plurality of fingerprint arrays stored in metadata of a respective similarity group, wherein each fingerprint array comprises a fingerprint indexed to a plurality of corresponding segment sizes;
access, by the worker node, a compression region name in the metadata of the respective similarity group, the accessed compression region name associated with the particular fingerprint array;
identify, by the worker node, a corruption of the respective similarity group based on the temporary hash score failing to match the accessed compression region name, the similarity group corruption indicating one or more portions of the particular fingerprint array in the metadata of the respective similarity group have been improperly modified.
13. The system of claim 12 , wherein the plurality of instructions, when executed, further cause the one or more processors to:
call, by the worker node, object storage according to the accessed compression region name in response to determining that the temporary hash score matches the accessed compression region name; and
identify, by the worker node, the compression region corruption on a failure to retrieve a compression region in object storage identified by the accessed compression region name.
14. The system of claim 8 , wherein the plurality of instructions, when executed, further cause the one or more processors to:
send, by the worker node, to the controller module, by the worker node, a message indicating completion of all attempts for identifying existence of respective slice recipe and similarity group corruption verifications.
15. A computer program product comprising a non-transitory computer-readable medium having a computer-readable program code embodied therein to be executed by one or more processors, the program code including instructions to:
store, by a worker node, at least one slice recipe into worker node memory from object storage, the at least one loaded slice recipe corresponding to a slice recipe name received from a controller module;
receive, by the worker node, one or more similarity group identifiers from the controller module;
detect, by the worker node:
(i) whether a slice recipe corruption exists for each slice recipe in worker node memory based on a comparison between metadata of the respective slice recipe and metadata of a corresponding similarity group identified by the controller module,
(ii) whether a similarity group corruption exists based on a comparison between a first fingerprint array and a compression region name in the metadata of the corresponding similarity group, and
(iii) whether a compression region corruption exists based on a comparison between the compression region name and object storage; and
send, by the worker node, one or more instances of slice recipe, similarity group and compression region corruptions to the controller module.
16. The computer program product of claim 15 , the program code including further instructions to:
receive, by the worker node, one or more slice recipe names identified by the controller module as needing to be verified, the one or more identified slice recipe names corresponding to respective similarity groups pre-assigned to the worker node;
load into worker node memory, by the worker node, slice recipe from object storage that corresponds to the slice recipe name;
access, by the worker node, a similarity group identifier in the loaded slice recipe; and
store, by the worker node, the loaded slice recipe in a slice recipe list associated with the similarity group identifier.
17. The computer program product of claim 16 , the program code including further instructions to:
sort, by the worker node, the slice recipe, that were loaded into worker memory, into slice recipe lists associated with a respective similarity group identifiers; and
send, by the worker node, a completion message to the controller module notifying the controller module that sorting slice recipes is complete.
18. The computer program product of claim 15 , the program code including further instructions to:
identify, by the worker node, a corruption of a respective slice recipe based on detecting a fingerprint within a fingerprint array in metadata of the respective slice recipe does not match any fingerprints stored amongst a plurality of fingerprint arrays in metadata of a similarity group associated with the respective slice recipe, the fingerprint array including a fingerprint indexed to a plurality of corresponding segment sizes, the slice recipe corruption indicating that the fingerprint in the respective slice recipe that fails to match has been improperly modified.
19. The computer program product of claim 15 , the program code including further instructions to:
determine, by the worker node, a temporary hash score based on a particular fingerprint array from a plurality of fingerprint arrays stored in metadata of a respective similarity group, wherein each fingerprint array comprises a fingerprint indexed to a plurality of corresponding segment sizes;
access, by the worker node, a compression region name in the metadata of the respective similarity group, the accessed compression region name associated with the particular fingerprint array;
identify, by the worker node, a corruption of the respective similarity group based on the temporary hash score failing to match the accessed compression region name, the similarity group corruption indicating one or more portions of the particular fingerprint array in the metadata of the respective similarity group have been improperly modified.
20. The computer program product of claim 19 , the program code including further instructions to:
call, by the worker node, object storage according to the accessed compression region name in response to determining that the temporary hash score matches the accessed compression region name; and
identify, by the worker node, the compression region corruption on a failure to retrieve a compression region in object storage identified by the accessed compression region name.