Fast deduplication data verification
An information management system provides a data deduplication system that uses a primary table, a deduplication chunk table, and a chunk integrity table to ensure that a referenced deduplicated data block is only verified once during the data verification of a backup or other replication operation. The data deduplication system may reduce the computational and storage overhead associated with traditional data verification processes. The primary table, the deduplication chunk table, and the chunk integrity table, all of which may be stored in a deduplication database, can also ensure synchronization between the deduplication database and secondary storage devices.
1. A networked information management system configured to verify integrity of deduplication data, the networked information management system comprising:
one or more computing devices comprising computer hardware configured to:
retrieve a deduplication chunk table, wherein the deduplication chunk table identifies a first data chunk;
retrieve a first data file associated with the first data chunk, wherein the first data file comprises a plurality of containers that each store one or more data blocks;
for each container in the plurality of containers,
perform a data integrity verification of corresponding one or more data blocks of the respective container,
wherein the data integrity verification includes determining whether the corresponding one or more data blocks of the respective container is valid, and
store results of the data integrity verification of the corresponding one or more data blocks of the respective container in association with an entry corresponding to the first data chunk in a chunk integrity table;
receive a request to verify integrity of at least one or more data blocks referenced in the plurality of containers; and
for each data block of the at least one or more data blocks referenced in the plurality of containers for which a request to verify integrity is received,
determine that a first link in the first data chunk references the respective data block, and
verify integrity of the respective data block by performing a lookup of the chunk integrity table to determine whether a container in the plurality of containers that references the respective data block is verified.
2. The networked information management system of claim 1 , wherein the computer hardware is further configured to, for each container in the plurality of containers, store a zero bit in association with the entry corresponding to the first data chunk in the chunk integrity table in response to a determination that least one of the corresponding one or more data blocks of the respective container cannot be verified.
3. The networked information management system of claim 1 , wherein the computer hardware is further configured to, for each container in the plurality of containers, store a one bit in association with the entry corresponding to the first data chunk in the chunk integrity table in response to a determination that all of the corresponding one or more data blocks of the respective container can be verified.
4. The networked information management system of claim 1 , wherein the computer hardware is further configured to retrieve the deduplication chunk table in response to a request to verify data in a backup.
5. The networked information management system of claim 4 , wherein the computer hardware is further configured to cause the chunk integrity table to be deleted after verification of the data in the backup is complete.
6. The networked information management system of claim 1 , wherein the corresponding container in the plurality of containers is verified if the one or more data blocks stored in the corresponding container are also decompressible, and decryptable.
7. The networked information management system of claim 1 , wherein, for each container in the plurality of containers, the corresponding one or more data blocks of the respective container are referenced by one or more links in the first data chunk.
8. The networked information management system of claim 1 , wherein the stored values allow the networked information management system to verify the integrity of the data stored in the first data chunk without having to verify the one or more data blocks in the plurality of containers more than once.
9. The networked information management system of claim 1 , wherein the computer hardware is further configured to retrieve the corresponding first data file based on an index stored in the first data chunk.
10. The networked information management system of claim 1 , wherein the deduplication chunk table identifies the first data chunk and a set of other data chunks.
11. The networked information management system of claim 10 , wherein the computer hardware is further configured to:
perform a verification of other containers of other data files associated with the other data chunks; and
store results of the verification of the other containers in the chunk integrity table.
12. A computer-implemented method for verifying integrity of deduplication data, the computer-implemented method comprising:
retrieving a deduplication chunk table, wherein the deduplication chunk table identifies a first data chunk;
retrieving a first file associated with the first data chunk, wherein the first file comprises a plurality of containers that each store one or more data blocks;
for each container in the plurality of containers,
performing a data integrity verification of corresponding one or more data blocks of the respective container,
wherein the data integrity verification includes determining whether the corresponding one or more data blocks of the respective container is readable, and
storing a value representing a result of the data integrity verification of the corresponding one or more data blocks of the respective container in association with an entry corresponding to the first data chunk in a chunk integrity table;
identifying a request to verify integrity of at least one or more data blocks referenced in one or more containers in the plurality of the containers; and
for each data block of the at least one or more data blocks referenced in the one or more containers for which a request to verify integrity is identified,
determining that a first link in the first data chunk references the respective data block, and
verifying integrity of the respective data block by performing a lookup of the chunk integrity table to determine whether a container in the one or more containers that references the respective data block is verified.
13. The computer-implemented method of claim 12 , further comprising, for each container in the plurality of containers, storing a zero bit in association with the entry corresponding to the first data chunk in the chunk integrity table in response to a determination that least one of the corresponding one or more data blocks of the respective container cannot be verified.
14. The computer-implemented method of claim 12 , further comprising, for each container in the plurality of containers, storing a one bit in association with the entry corresponding to the first data chunk in the chunk integrity table in response to a determination that all of the corresponding one or more data blocks of the respective container can be verified.
15. The computer-implemented method of claim 12 , further comprising retrieving the deduplication chunk table in response to a request to verify data in a backup.
16. The computer-implemented method of claim 15 , further comprising causing the chunk integrity table to be deleted after verification of the data in the backup is complete.
17. The computer-implemented method of claim 12 , wherein the corresponding container in the plurality of containers is verified if the one or more data blocks stored in the corresponding container are also decompressible, and decryptable.
18. The computer-implemented method of claim 12 , wherein, for each container in the plurality of containers, the corresponding one or more data blocks of the respective container are referenced by one or more links in the first data chunk.
19. The computer-implemented method of claim 12 , further comprising retrieving the first data file based on an index stored in the first data chunk.
20. An information management system configured to verify integrity of deduplication data, the information management system comprising:
one or more computing devices comprising computer hardware configured to:
retrieve a first data file associated with a first data chunk, wherein the first data file comprises a plurality of containers that each store one or more data blocks;
for each container in the plurality of containers,
perform a data integrity verification of corresponding one or more data blocks of the respective container, wherein the data integrity verification includes determining whether the corresponding one or more data blocks of the respective container is valid, and
store results of the data integrity verification of the corresponding one or more data blocks of the respective container in a chunk integrity table;
receive a request to verify integrity of at least one or more data blocks referenced in the plurality of containers; and
for each data block of the at least one or more data blocks referenced in the plurality of containers for which a request to verify integrity is received,
verify integrity of the respective data block by performing a lookup of the chunk integrity table to determine whether a container in the plurality of containers that references the respective data block is verified.