Minimize damage caused by corruption of de-duplicated data
View Patent ↗An embodiment for minimizing damage caused by corruption of de-duplicated data is disclosed. One embodiment counts the number of references to a data segment indexed by a de-duplication index. When a reference count for the data segment reaches a first threshold a duplicate of the data segment is generated. In addition, the duplicate of the data segment is also stored to minimize damage of the de-duplicated data caused by a corruption of the data segment.
1. A method, comprising:
controlling a data de-duplication process to receive a data segment;
controlling the process to compute a de-duplication hash for the data segment;
controlling the process to access a de-duplication index, where the de-duplication index comprises keys and values, where the de-duplication index is configured to be accessed using the de-duplication hash as a key, and where the de-duplication index is configured to store references to unique data segments available to the process;
controlling the process to determine whether the de-duplication hash for the data segment is present in the de-duplication index;
upon determining that the de-duplication hash is not present in the de-duplication index:
controlling the process to store the data segment on a first storage device as a unique instance of the data segment;
controlling the process to store, in the de-duplication index, information sufficient to locate the data segment as stored on the first storage device; and
controlling the process to set to an initial value a reference count associated with the data segment;
and
upon determining that the de-duplication hash for the data segment is present in the de-duplication index:
controlling the process to update the reference count associated with the data segment; and
upon determining that the reference count associated with the data segment exceeds a threshold:
controlling the process to store the data segment on a second storage device as an additional instance of the data segment; where controlling the process to update the reference count includes selectively not updating the reference count if the data segment was not received within a threshold period of time after the receipt of a previous instance of the data segment; and
controlling the computer to store, in the de-duplication index, information sufficient to locate the data segment as stored on the second storage device.
2. The method of claim 1 , where the data segment is a variable length segment whose boundaries are determined, at least in part, by the de-duplication process using a rolling hash.
3. The method of claim 1 ,where the de-duplication hash is a cryptographic hash.
4. The method of claim 1 , where the first storage device is one of, a hard disk drive, a solid state drive, a non-volatile random access memory, a tape drive, and a non-volatile solid state storage device.
5. The method of claim 4 , where the second storage device is a different storage device than the first storage device, and where the second storage device is one of, a hard disk drive, a solid state drive a non-volatile random access memory, a tape drive, and a non-volatile solid state storage device.
6. The method of claim 1 , where the threshold is greater than the initial value, and where the threshold is a function of the reference count.
7. The method of claim 1 , where controlling the process to store, in the de-duplication index, information sufficient to locate the data segment as stored on the first storage device comprises storing in a value field located using the de-duplication hash as a key, a reference to the data segment.
8. The method of claim 7 , where controlling the process to store in the de-duplication index, information sufficient to locate the data segment as stored on the second storage device comprises storing, in the value field, a second, different reference to the data segment.