Testing verification paths provided by a verification microservice for a deduplicated object storage system
Described is a system for detecting corruption in a deduplicated object storage system accessible by one or more microservices while minimizing costly read operations on objects. A testing tool inserts object corruptions in object storage. A verification path is performed by a controller module and one or more worker nodes. The testing tool verifies whether the executed verification path accurately detected the testing tool's inserted object corruptions.
1. A computer-implemented method, comprising:
uploading one or more objects into object storage of a deduplicated storage system, the objects having one or more of an object recipe name, an object recipe, a slice recipe name, a slice recipe, a similarity group name and a similarity group;
corrupting metadata in respective subsets of the uploaded objects;
performing a slice recipe verification path or a similarity group verification path over the object storage;
receiving a corrupted objects list describing one or more corruptions identified by a controller module and one or more worker nodes during performance of a respective verification path; and
confirming whether the identified corruptions in the corrupted objects list resulting from performance of the respective verification path correspond with the corrupted metadata in the respective subsets of the uploaded objects.
2. The computer-implemented method of claim 1 , wherein corrupting metadata in respective subsets of the uploaded objects comprises:
generating a first subset of corrupted versions of the uploaded objects by corrupting internal object metadata of a first subset of the uploaded objects, the internal object metadata referencing data internal to an uploaded object;
generating a second subset of corrupted versions of the uploaded objects by corrupting external object metadata of a second subset of the uploaded objects, the external object metadata providing a reference to another uploaded object, wherein the first and second subsets of uploaded objects may share at least one common uploaded object;
deleting a third subset of the uploaded objects; and
creating a local mapping identifying each respective corrupted object version.
3. The computer-implemented method of claim 2 , wherein confirming whether the identified corruptions in the corrupted objects list resulting from performance of the respective verification path correspond with the corrupted metadata in the respective subsets of the uploaded objects comprises:
receiving the corrupted objects list based on completion of the slice recipe verification path or the similarity group verification path over the object storage;
comparing the local mapping to the corrupted objects list; and
determining the performed verification path executed correctly based on the corrupted objects list matching the local mapping.
4. The computer-implemented method of claim 2 , wherein deleting a third subset of the uploaded objects comprises:
removing each uploaded objected that has a respective corrupted version from object storage; and
replacing each removed uploaded objected in object storage with its respective corrupted version.
5. The computer-implemented method of claim 1 , wherein corrupting metadata in respective subsets of the uploaded objects comprises at least one of:
corrupting at least one of a slice part and a slice count stored in a data structure of an object recipe, wherein the slice part comprises internal metadata of the object recipe.
6. The computer-implemented method of claim 1 , wherein corrupting metadata in respective subsets of the uploaded objects comprises at least one of:
corrupting at least one fingerprint and a similarity group reference stored within a data structure of a slice recipe; and
corrupting similarity group identifier in a slice recipe string name of the slice recipe.
7. The computer-implemented method of claim 1 , wherein corrupting metadata in respective subsets of the uploaded objects comprises at least one of:
corrupting at least one compression region name and at least one fingerprint stored within a data structure of a similarity group.
8. A system comprising:
one or more processors; and
a non-transitory computer readable medium storing a plurality of instructions, which when executed, cause the one or more processors to:
upload one or more objects into object storage of a deduplicated storage system, the objects having one or more of an object recipe name, an object recipe, a slice recipe name, a slice recipe, a similarity group name and a similarity group; corrupt metadata in respective subsets of the uploaded objects;
perform a slice recipe verification path or a similarity group verification path over the object storage;
receive a corrupted objects list describing one or more corruptions identified by a controller module and one or more worker nodes during performance of a respective verification path; and
confirm whether the identified corruptions in the corrupted objects list resulting from performance of the respective verification path correspond with the corrupted metadata in the respective subsets of the uploaded objects.
9. The system of claim 8 , wherein corrupt metadata in respective subsets of the uploaded objects comprises:
generate a first subset of corrupted versions of the uploaded objects by corrupting internal object metadata of a first subset of the uploaded objects, the internal object metadata referencing data internal to an uploaded object;
generate a second subset of corrupted versions of the uploaded objects by corrupting external object metadata of a second subset of the uploaded objects, the external object metadata providing a reference to another uploaded object, wherein the first and second subsets of uploaded objects may share at least one common uploaded object;
delete a third subset of the uploaded objects; and
create a local mapping identifying each respective corrupted object version.
10. The system of claim 9 , wherein confirm whether the identified corruptions in the corrupted objects list resulting from performance of the respective verification path correspond with the corrupted metadata in the respective subsets of the uploaded objects comprises:
receive the corrupted objects list based on completion of the slice recipe verification path or the similarity group verification path over the object storage;
compare the local mapping to the corrupted objects list; and
determine the performed verification path executed correctly based on the corrupted objects list matching the local mapping.
11. The system of claim 9 , wherein delete a third subset of the uploaded objects comprises:
remove each uploaded objected that has a respective corrupted version from object storage; and
replace each removed uploaded objected in object storage with its respective corrupted version.
12. The system of claim 8 , wherein corrupt metadata in respective subsets of the uploaded objects comprises at least one of:
corrupt at least one of a slice part and a slice count stored in a data structure of an object recipe, wherein the slice part comprises internal metadata of the object recipe.
13. The system of claim 8 , wherein corrupt metadata in respective subsets of the uploaded objects comprises at least one of:
corrupt at least one fingerprint and a similarity group reference stored within a data structure of a slice recipe; and
corrupt similarity group identifier in a slice recipe string name of the slice recipe.
14. The system of claim 8 , wherein corrupt metadata in respective subsets of the uploaded objects comprises at least one of:
corrupt at least one compression region name and at least one fingerprint stored within a data structure of a similarity group.
15. A computer program product comprising a non-transitory computer-readable medium having a computer-readable program code embodied therein to be executed by one or more processors, the program code including instructions to:
upload one or more objects into object storage of a deduplicated storage system, the objects having one or more of an object recipe name, an object recipe, a slice recipe name, a slice recipe, a similarity group name and a similarity group;
corrupt metadata in respective subsets of the uploaded objects;
perform a slice recipe verification path or a similarity group verification path over the object storage;
receive a corrupted objects list describing one or more corruptions identified by a controller module and one or more worker nodes during performance of a respective verification path; and
confirm whether the identified corruptions in the corrupted objects list resulting from performance of the respective verification path correspond with the corrupted metadata in the respective subsets of the uploaded objects.
16. The computer program product of claim 15 , wherein corrupt metadata in respective subsets of the uploaded objects comprises:
generate a first subset of corrupted versions of the uploaded objects by corrupting internal object metadata of a first subset of the uploaded objects, the internal object metadata referencing data internal to an uploaded object;
generate a second subset of corrupted versions of the uploaded objects by corrupting external object metadata of a second subset of the uploaded objects, the external object metadata providing a reference to another uploaded object, wherein the first and second subsets of uploaded objects may share at least one common uploaded object;
delete a third subset of the uploaded objects; and
create a local mapping identifying each respective corrupted object version.
17. The computer program product of claim 16 , wherein confirm whether the identified corruptions in the corrupted objects list resulting from performance of the respective verification path correspond with the corrupted metadata in the respective subsets of the uploaded objects comprises:
receive the corrupted objects list based on completion of the slice recipe verification path or the similarity group verification path over the object storage;
compare the local mapping to the corrupted objects list; and
determine the performed verification path executed correctly based on the corrupted objects list matching the local mapping.
18. The computer program product of claim 15 , wherein delete a third subset of the uploaded objects comprises:
remove each uploaded objected that has a respective corrupted version from object storage; and
replace each removed uploaded objected in object storage with its respective corrupted version.
19. The computer program product of claim 15 , wherein corrupt metadata in respective subsets of the uploaded objects comprises at least one of:
corrupt at least one of a slice part and a slice count stored in a data structure of an object recipe, wherein the slice part comprises internal metadata of the object recipe.
20. The computer program product of claim 15 , wherein corrupt metadata in respective subsets of the uploaded objects comprises at least one of:
corrupt at least one fingerprint and a similarity group reference stored within a data structure of a slice recipe; and
corrupt similarity group identifier in a slice recipe string name of the slice recipe.
21. The computer program product of claim 15 , wherein corrupt metadata in respective subsets of the uploaded objects comprises at least one of:
corrupt at least one compression region name and at least one fingerprint stored within a data structure of a similarity group.