Data validation of container objects in a deduplication storage system

1 . A computing device comprising:

at least one processor;

a memory; and

at least one machine-readable storage medium comprising instructions executable by the at least one processor to:

receive a data unit to be stored in a persistent storage of a deduplication storage system;

identify a container index that includes metadata regarding the data unit;

identify a first container entity group (CEG) object that is recorded, in the container index, as a deduplicated storage location of the data unit;

in response to a determination that a validation state of the first CEG object is recorded as invalid in the container index, store the data unit in a second CEG object; and

modify the container index to record the second CEG object as the deduplicated storage location of the data unit.

2 . The computing device of claim 1 , including instructions executable by the at least one processor to, in response to the determination that the validation state of the first CEG object is recorded as invalid in the container index:

increment, in the container index, a reference count for the data unit.

3 . The computing device of claim 2 , including instructions executable by the at least one processor to, in response to a determination that the validation state of the first CEG object is recorded as unknown in the container index:

load the first CEG object from a remote storage into the memory;

perform one or more content test on the first CEG object loaded in the memory; and

determine whether the first CEG object includes correct data based on the one or more content tests.

4 . The computing device of claim 3 , including instructions executable by the at least one processor to, in response to a determination that the first CEG object includes the correct data based on the one or more content tests:

record the validation state of the first CEG object as valid in the container index; and

increment, in the container index, the reference count for the data unit.

5 . The computing device of claim 3 , including instructions executable by the at least one processor to, in response to a determination that the first CEG object does not include the correct data based on the one or more content tests:

record the validation state of the first CEG object as invalid in the container index;

store the data unit in the second CEG object;

modify the container index to record the second CEG object as the deduplicated storage location of the data unit; and

increment, in the container index, the reference count for the data unit.

6 . The computing device of claim 1 , including instructions executable by the at least one processor to:

detect a system validation trigger event for a group of CEG objects stored in the persistent storage of the deduplication storage system; and

in response to a detection of the system validation trigger event, for each of the group of CEG objects that has a validation state recorded as valid, change the validation state to be recorded as unknown.

7 . The computing device of claim 1 , including instructions executable by the at least one processor to:

detect a random validation trigger event for a third CEG object with a validation state that is recorded as valid; and

in response to a detection of the random validation trigger event, change the validation state of the third CEG object to be recorded as unknown.

8 . The computing device of claim 1 , including instructions executable by the at least one processor to:

detect a match of a received data unit to a fourth CEG object with a validation state that is recorded as valid;

in response to a detection of the match of the received data unit to the fourth CEG object, determine an accumulated number of matches since the validation state of the fourth CEG object was recorded as valid;

in response to a determination that the accumulated number of matches exceeds a predefined threshold, change the validation state of the fourth CEG object to be recorded as unknown; and

reset the accumulated number of matches for the fourth CEG object to zero.

9 . A method comprising:

receiving, by a storage controller of a deduplication storage system, a data unit to be stored in a persistent storage of the deduplication storage system;

identifying, by the storage controller, a container index that includes metadata regarding the data unit;

identifying, by the storage controller, a first container entity group (CEG) object that is recorded, in the container index, as a deduplicated storage location of the data unit;

determining, by the storage controller, whether a validation state of the first CEG object is recorded in the container index;

in response to a determination that that a validation state of the first CEG object is recorded as invalid in the container index, storing, by the storage controller, the data unit in a second CEG object; and

modifying, by the storage controller, the container index to record the second CEG object as the deduplicated storage location of the data unit.

10 . The method of claim 9 , comprising, in response to the determination that the validation state of the first CEG object is recorded as invalid in the container index:

incrementing, in the container index, a reference count for the data unit.

11 . The method of claim 10 , comprising, in response to a determination that the validation state of the first CEG object is recorded as unknown in the container index:

loading the first CEG object from a remote storage into the memory;

performing one or more content test on the first CEG object loaded in the memory; and

determining whether the first CEG object includes correct data based on the one or more content tests.

12 . The method of claim 11 , comprising, in response to a determination that the first CEG object includes the correct data based on the one or more content tests:

recording the validation state of the first CEG object as valid in the container index; and

incrementing, in the container index, the reference count for the data unit.

13 . The method of claim 11 , comprising, in response to a determination that the first CEG object does not include the correct data based on the one or more content tests:

recording the validation state of the first CEG object as invalid in the container index;

storing the data unit in the second CEG object;

modifying the container index to record the second CEG object as the deduplicated storage location of the data unit; and

incrementing, in the container index, the reference count for the data unit.

14 . The method of claim 9 , comprising:

detecting a system validation trigger event for a group of CEG objects stored in the persistent storage of the deduplication storage system; and

in response to detecting the system validation trigger event, for each of the group of CEG objects that has a validation state recorded as valid, changing the validation state to be recorded as unknown.

15 . A non-transitory machine-readable storage medium comprising instructions executable by at least one processor to:

receive a data unit to be stored in a persistent storage of a deduplication storage system;

identify a container index that includes metadata regarding the data unit;

identify a first container entity group (CEG) object that is recorded, in the container index, as a deduplicated storage location of the data unit;

in response to a determination that a validation state of the first CEG object is recorded as invalid in the container index, store the data unit in a second CEG object; and

modify the container index to record the second CEG object as the deduplicated storage location of the data unit.

16 . The non-transitory machine-readable medium of claim 15 , including instructions executable by the at least one processor to, in response to the determination that the validation state of the first CEG object is recorded as invalid in the container index:

increment, in the container index, a reference count for the data unit.

17 . The non-transitory machine-readable medium of claim 16 , including instructions executable by the at least one processor to, in response to a determination that the validation state of the first CEG object is recorded as unknown in the container index:

load the first CEG object from a remote storage into the memory;

perform one or more content test on the first CEG object loaded in the memory; and

determine whether the first CEG object includes correct data based on the one or more content tests.

18 . The non-transitory machine-readable medium of claim 17 , including instructions executable by the at least one processor to, in response to a determination that the first CEG object includes the correct data based on the one or more content tests:

record the validation state of the first CEG object as valid in the container index; and

increment, in the container index, the reference count for the data unit.

19 . The non-transitory machine-readable medium of claim 17 , including instructions executable by the at least one processor to, in response to a determination that the first CEG object does not include the correct data based on the one or more content tests:

record the validation state of the first CEG object as invalid in the container index;

store the data unit in the second CEG object;

modify the container index to record the second CEG object as the deduplicated storage location of the data unit; and

increment, in the container index, the reference count for the data unit.

20 . The non-transitory machine-readable medium of claim 15 , including instructions executable by the at least one processor to:

detect a random validation trigger event for a third CEG object with a validation state that is recorded as valid; and

in response to a detection of the random validation trigger event, change the validation state of the third CEG object to be recorded as unknown.