Mutable data objects content verification tool
Data object content verification systems and processes provide perfect reliability and low storage overhead. Object data is generated in a reproducible manner based upon object locally stored object metadata. The object data is stored to an object storage system. The stored object data is subsequently verified by retrieving the object metadata, reproducing the original object data, and comparing the stored and original object data. The data object content verification systems and processes support both mutable and immutable data object content verification.
1. A method for use with a distributed object storage system, the method comprising:
generating, in a processor, a plurality of object IDs and corresponding original object metadata;
for each object ID, generating corresponding original object data in a reproducible manner based upon the corresponding original object metadata, the original object data generated based upon a first object size;
storing the corresponding original object metadata to a metadata storage device and writing the corresponding original object data to the distributed object storage system;
selecting a plurality of the object IDs of the original object data to be mutated; and
for the object IDs to be mutated:
generating corresponding mutated object data in a reproducible manner based upon corresponding mutated object metadata, the mutated object data generated based upon a second object size that is greater than the first object size;
storing the corresponding mutated object metadata to the metadata storage device and writing the mutated object data to the distributed object storage system;
retrieving the object ID and mutated object metadata from the metadata storage device;
reproducing the mutated object data using the mutated object metadata; and
comparing the reproduced mutated object data to the retrieved mutated object metadata to identify corruption in the distributed object storage system.
2. The method of claim 1 wherein mutated object metadata comprises an updated object ID and a bitmask.
3. The method of claim 2 wherein reproducing the mutated object data using the mutated object metadata comprises:
reproducing original object data using the original object metadata;
generating updated object data using the updated object ID; and
combining the reproduced original data and the updated object data according to the bitmask to reproduce the mutated object data.
4. The method of claim 2 wherein mutated object metadata comprises an ordered list of updated object ID and bitmask tuples.
5. The method of claim 4 wherein reproducing the mutated object data using the mutated object metadata comprises:
reproducing original object data using the original object metadata;
for each tuple in the ordered list of updated object ID and bitmask tuples, iteratively generating updated object data using the updated object ID and combining the reproduced original data and the updated object data according to the bitmask to generate the reproduce the mutated object data.
6. A system for use with a distributed object storage system, the system comprising:
a content generator configured to generate object data in a reproducible manner based upon corresponding object metadata, the original object data generated based upon a first object size;
an object creator configured to:
generate a plurality of object IDs and corresponding original object metadata;
for each object ID, generate corresponding original object data using the content generator and the corresponding original object metadata;
store the corresponding original object metadata to a metadata storage device; and
write the corresponding original object data to the distributed object storage system;
an object mutator configured to:
select a plurality of the object IDs of the original object data to be mutated; and
for the object IDs to be mutated:
generate corresponding mutated object data in a reproducible manner using the content generator and the mutated object metadata, the mutated object data generated based upon a second object size that is greater than the first object size;
store the corresponding mutated object metadata to the metadata storage device; and
write the mutated object data to the distributed object storage system; and
a content verifier configure to:
for the object IDs to be mutated:
retrieve the object ID and mutated object metadata from the metadata storage device;
reproduce the mutated object data using the mutated object metadata; and
compare the reproduced mutated object data to the retrieved mutated object metadata to identify corruption in the distributed object storage system.
7. The system of claim 6 wherein mutated object metadata comprises an updated object ID and a bitmask.
8. The system of claim 7 wherein the content verifier is configured to:
reproduce original object data using the original object metadata;
generate updated object data using the updated object ID; and
combine the reproduced original data and the updated object data according to the bitmask to reproduce the mutated object data.
9. The system of claim 7 wherein mutated object metadata comprises an ordered list of updated object ID and bitmask tuples.
10. The system of claim 9 wherein the content verifier is configured to:
reproduce original object data using the original object metadata;
for each tuple in the ordered list of updated object ID and bitmask tuples, iteratively generate updated object data using the updated object ID and combine the reproduced original data and the updated object data according to the bitmask to generate the reproduce the mutated object data.