Verification microservice for a deduplicated object storage system
Described is a system for detecting corruption in a deduplicated object storage system accessible by one or more microservices while minimizing costly read operations on objects. A similarity group verification path is selected based upon detection of an object storage memory size condition. The similarity group verification path is performed by one or more worker nodes. The similarity group verification path includes worker node phases that verify whether objects have been corrupted without having to incur costly read operations.
1. A computer-implemented method, comprising:
identifying, by a worker node, a similarity group loaded into a memory that stores the worker node, the similarity group comprising a first plurality of arrays of fingerprints and corresponding segment sizes in metadata, each array having an associated compression region name based on a hash value produced by a hash of the respective array;
determining, by the worker node, whether a hash value produced by a hash of a first array of the first plurality of arrays matches any compression region name in the metadata of the similarity group;
identifying, by the worker node, a slice recipe loaded into the memory that stores the worker node, the slice recipe comprising a second plurality of arrays of fingerprints and corresponding segment sizes in metadata;
determining, by the worker node, whether a second array, of the second plurality of arrays of fingerprints and corresponding segment sizes in the metadata of the slice recipe, matches the first array in the metadata of the similarity group stored in the memory of the worker node; and
tracking, by the worker node, instances of corruption, based on whether: i) the hash value of the first array fails to match any of the compression region names of the first plurality of arrays in the metadata of the similarity group, or ii) the first array fails to match any of the second plurality of arrays in the metadata of the slice recipe to be sent to a controller.
2. The computer-implemented method of claim 1 , further comprising:
calculating, by the worker node, a hash score based on the first array in the metadata of the similarity group;
identifying, by the worker node, a compression region name in the metadata of the similarity group associated with the first array;
comparing, by the worker node, the compression region name and the hash score; and
determining, by the worker node, an existence of corruption based on a failure to match between the compression region name and the hash score.
3. The computer-implemented method of claim 2 , further comprising:
verifying, by the worker node, whether the compression region name exists in object storage; and
determining, by the worker node, an existence of corruption based on an absence of the compression region name in the object storage.
4. The computer-implemented method of claim 1 , further comprising:
receiving, by the worker node, one or more slice recipe names identified by the controller as needing to be verified; and
loading, by the worker node, from object storage into the memory of the worker node, a slice recipe that corresponds to each respective identified slice recipe name.
5. The computer-implemented method of claim 4 , further comprising, for each slice recipe loaded into the memory of the worker node:
comparing, by the worker node, each entry in the first array to a corresponding entry in the second array;
determining, by the worker node, an existence of corruption based on a failure to match between compared entries in the first and the second arrays;
releasing, by the worker node, the slice recipe from the memory of the worker node upon completion of the comparison of all entries in the first and the second arrays; and
sending to the controller, by the worker node, a message based on the worker node completing verification of all slice recipe names identified by the controller.
6. The computer-implemented method of claim 1 , further comprising:
receiving, by the worker node, identification of one or more similarity groups by the controller as needing to be verified; and
sending, by the worker node, a message to the controller based on the worker node completing verification of all similarity groups identified by the controller as needing to be verified.
7. A system comprising:
one or more processors; and
a non-transitory computer readable medium storing a plurality of instructions, which when executed, cause the one or more processors to:
identify, by a worker node, a similarity group loaded into a memory that stores the worker node, the similarity group comprising a first plurality of arrays of fingerprints and corresponding segment sizes in metadata, each array having an associated compression region name based on a hash value produced by a hash of the respective array;
determine, by the worker node, whether a first hash value produced by a hash of a first array of the first plurality of arrays matches any compression region name in the metadata of the similarity group;
identify, by the worker node, a slice recipe loaded into the memory that stores the worker node, the slice recipe comprising a second plurality of arrays of fingerprints and corresponding segment sizes in metadata;
determine, by the worker node, whether a second array, of the second plurality of arrays of fingerprints and corresponding segment sizes in the metadata of the slice recipe, matches the first array in the metadata of the similarity group stored in the memory of the worker node; and
track, by the worker node, instances of corruption, based on whether: i) the hash value of the first array fails to match any of the compression region names of the first plurality of arrays in the metadata of the similarity group, or ii) the first array fails to match any of the second plurality of arrays in the metadata of the slice recipe to be sent to a controller.
8. The system of claim 7 , wherein the plurality of instructions, when executed, further cause the one or more processors to:
calculate, by the worker node, a hash score based on the first array in the metadata of the similarity group;
identify, by the worker node, a compression region name in the metadata of the similarity group associated with the first array;
compare, by the worker node, the compression region name and the hash score; and
determine, by the worker node, an existence of corruption based on a failure to match between the compression region name and the hash score.
9. The system of claim 8 , wherein the plurality of instructions, when executed, further cause the one or more processors to:
verify, by the worker node, whether the compression region name exists in object storage; and
determine, by the worker node, an existence of corruption based on an absence of the compression region name in the object storage.
10. The system of claim 7 , wherein the plurality of instructions, when executed, further cause the one or more processors to:
receive, by the worker node, one or more slice recipe names identified by the controller as needing to be verified; and
load, by the worker node, from object storage into the memory of the worker node, a slice recipe that corresponds to each respective identified slice recipe name.
11. The system of claim 10 , wherein the plurality of instructions, when executed, further cause the one or more processors, for each slice recipe loaded into worker memory, to:
compare, by the worker node, each entry in the first array to a corresponding entry in the second array;
determine, by the worker node, an existence of corruption based on a failure to match between compared entries in the first and the second arrays;
release, by the worker node, the slice recipe from the memory of the worker node upon completion of the comparison of all entries in the first and the second arrays; and
send to the controller, by the worker node, a message based on the worker node completing verification of all slice recipe names identified by the controller.
12. The system of claim 7 , wherein the plurality of instructions, when executed, further cause the one or more processors to:
receive, by the worker node, identification of one or more similarity groups by the controller as needing to be verified; and
send, by the worker node, a message to the controller based on the worker node completing verification of all similarity groups identified by the controller as needing to be verified.
13. A computer program product comprising a non-transitory computer-readable medium having a computer-readable program code embodied therein to be executed by one or more processors, the program code including instructions to:
identify, by a worker node, a similarity group loaded into a memory that stores the worker node, the similarity group comprising a first plurality of arrays of fingerprints and corresponding segment sizes in metadata, each array having an associated compression region name based on a hash value produced by a hash of the respective array;
determine, by the worker node, whether a first hash value produced by a hash of a first array of the first plurality of arrays matches any compression region name in the metadata of the similarity group;
identify by the worker node, a slice recipe loaded into the memory that stores the worker node, the slice recipe comprising a second plurality of arrays of fingerprints and corresponding segment sizes in metadata;
determine, by the worker node, whether a second array, of the second plurality of arrays of fingerprints and corresponding segment sizes in the metadata of the slice recipe matches the first array in the metadata of the similarity group stored in the memory of the worker node; and
track by the worker node, instances of corruption, based on whether: i) the hash value of the first array fails to match any of the compression region names of the first plurality of arrays in the metadata of the similarity group, or ii) the first array fails to match any of the second plurality of arrays in the metadata of the slice recipe to be sent to a controller.
14. The computer program product of claim 13 , wherein the program code includes further instructions to:
calculate, by the worker node, a hash score based on the first array in the metadata of the similarity group;
identify, by the worker node, a compression region name in the metadata of the similarity group associated with the first array;
compare, by the worker node, the compression region name and the hash score; and
determine, by the worker node, an existence of corruption based on an inconsistency a failure to match between the compression region name and the hash score.
15. The computer program product of claim 14 , wherein the program code includes further instructions to:
verify, by the worker node, whether the compression region name exists in object storage; and
determine, by the worker node, an existence of corruption based on an absence of the compression region name in the object storage.
16. The computer program product of claim 13 , wherein the program code includes further instructions to:
receive, by the worker node, one or more slice recipe names identified by the controller as needing to be verified; and
load, by the worker node, from object storage into the memory of the worker node, a slice recipe that corresponds to each respective identified slice recipe name.
17. The computer program product of claim 16 , wherein the program code includes further instructions to, for each slice recipe loaded into worker memory:
compare, by the worker node, each entry in the first array to a corresponding entry in the second array;
determine, by the worker node, an existence of corruption based on a failure to match between compared entries in the first and the second arrays;
release, by the worker node, the slice recipe from the memory of the worker node upon completion of the comparison of all entries in the first and the second arrays; and
send to the controller, by the worker node, a message based on the worker node completing verification of all slice recipe names identified by the controller.
18. The computer program product of claim 13 , wherein the program code includes further instructions to:
receive, by the worker node, identification of one or more similarity groups by the controller as needing to be verified; and
send, by the worker node, a message to the controller based on the worker node completing verification of all similarity groups identified by the controller as needing to be verified.