System and method for maintaining data integrity of data on a storage device
Systems and methods are disclosed herein for maintaining data integrity of data in a storage device. An exemplary method comprises determining whether checksums associated with data on the storage device are trusted or untrusted, responsive to determining that the checksums are trusted, retrieving the checksums from a checksum archive, otherwise initializing the checksums as unknown, when a received storage request is a write request, calculating a checksum of the data and updating the checksum in memory, when a received storage request is a read request and an in-memory checksum associated with the identified block is not unknown, calculating the checksum of the data in the identified block and comparing the checksum with an in-memory checksum associated with the identified block, determining that corruption has occurred when the checksums don't match and saving checksums associated with each of the uncorrupted data blocks of the storage device to an archive.
1. A method for maintaining data integrity of data in a storage device comprising:
determining whether checksums stored in a checksum archive and associated with data on the storage device mounted on a computer system are trusted or untrusted, wherein the checksum archive is on a different storage device;
responsive to determining that the checksums in the checksum archive are trusted, retrieving the checksums from the checksum archive, otherwise initializing the checksums as unknown;
detecting a storage request for an identified block of the storage device;
when the storage request is a write request, calculating a checksum of the data in the identified block and updating the checksum in memory of the computer system;
when the storage request is a read request and an in-memory checksum associated with the identified block is not unknown, i) calculating the checksum of the data in the identified block and comparing the calculated checksum with an in-memory checksum associated with the identified block, and ii) determining that a data corruption has occurred when the calculated checksum does not match the in-memory checksum; and
saving checksums associated with each of the data blocks of the storage device where the data corruption has not occurred to the checksum archive.
2. The method of claim 1 , further comprising:
wherein determining whether the checksums are trusted or untrusted comprises determining that the storage device was not turned off correctly;
detecting that there are checksums that are not trusted associated with blocks in the storage device; and
setting the untrusted checksums as unknown.
3. The method of claim 1 , further comprising:
calculating checksums for all blocks of the storage device using a background process for those checksums that are set as unknown.
4. The method of claim 1 , further comprising:
managing a bitmap containing a status for each block of the storage device; and
marking a status of a block as dirty if data stored in the block is changed during operation of the computer system; and
calculating checksums for blocks with a dirty status; and
storing the calculated checksums in memory of the computer system.
5. The method of claim 1 , further comprising:
when the storage request is a write request, calculating the checksum of the data asynchronously, after the data is written to the block of the storage device;
saving the checksums to the checksum archive upon receiving an unmount request to unmount the storage device from the computer system.
6. The method of claim 1 , further comprising:
storing the checksum in memory using write-through caching mode.
7. The method of claim 1 , wherein the checksum archive is a separate block device with a file system stored thereon with indexing or a local or remote key-value store with non-indexed search, wherein indexing refers to storing checksums in a linear array and finding the checksums for a corresponding block using a block number of the block.
8. The method of claim 7 , further comprising:
searching the local or remote key-value store using a non-indexed search comprising one of radix tree or hash table.
9. The method of claim 1 , wherein determining whether checksums in a checksum archive associated with data on the storage device mounted on a computer system are trusted or untrusted further comprises:
checking a trust flag, wherein the trust flag is stored on one of:
a file system mounted on the storage device;
a block of the storage device; or
the checksum archive.
10. The method of claim 1 , wherein, when the storage request is a write request, calculating a checksum of the data in the identified block either synchronously or a synchronously.
11. A system for maintaining data integrity of data in a storage device comprising:
a hardware processor configured to:
determine whether checksums stored in a checksum archive and associated with data on the storage device mounted on a computer system are trusted or untrusted, wherein the checksum archive is on a different storage device;
responsive to determining that the checksums in the checksum archive are trusted, retrieve the checksums from the checksum archive, otherwise initializing the checksums as unknown;
detect a storage request for an identified block of the storage device;
when the storage request is a write request, calculate a checksum of the data in the identified block and updating the checksum in memory of the computer system;
when the storage request is a read request and an in-memory checksum associated with the identified block is not unknown, i) calculate the checksum of the data in the identified block and comparing the calculated checksum with an in-memory checksum associated with the identified block and ii) determine that a data corruption has occurred when the calculated checksum does not match the in-memory checksum; and
save checksums associated with each of the data blocks of the storage device where the data corruption has not occurred to the checksum archive.
12. The system of claim 11 , wherein determining whether the checksums are trusted or untrusted comprises determining that the storage device was not turned off correctly;
detecting that there are checksums that are not trusted associated with blocks in the storage device; and
setting the untrusted checksums as unknown.
13. The system of claim 11 , the hardware processor further configured to:
calculate checksums for all blocks of the storage device using a background process for those checksums that are set as unknown.
14. The system of claim 11 , the hardware processor further configured to:
manage a bitmap containing a status for each block of the storage device; and
mark a status of a block as dirty if data stored in the block is changed during operation of the computer system; and
calculate checksums for blocks with a dirty status; and
store the calculated checksums in memory of the computer system.
15. The system of claim 11 , the hardware processor further configured to:
when the storage request is a write request, calculate the checksum of the data asynchronously, after the data is written to the block of the storage device;
save the checksums to the checksum archive upon receiving an unmount request to unmount the storage device from the computer system.
16. The system of claim 11 , the hardware processor further configured to:
store the checksum in memory using write-through caching mode.
17. The system of claim 11 , wherein the checksum archive is one of a separate block device with a file system stored thereon with indexing or a local or remote key-value store with non-indexed search, wherein indexing refers to storing checksums in a linear array and finding the checksums for a corresponding block using a block number of the block.
18. The system of claim 17 , the hardware processor further configured to:
search the local or remote key-value store using a non-indexed search comprising one of radix tree or hash table.
19. The system of claim 11 , wherein when the hardware processor determines whether checksums in a checksum archive associated with data on the storage device mounted on a computer system are trusted or untrusted, the hardware processor is further configured to:
check a trust flag, wherein the trust flag is stored on one of:
a file system mounted on the storage device;
a block of the storage device; or
the checksum archive.
20. A non-transitory computer-readable medium, storing computer-executable instructions thereon for executing, on a computing system, a computer process for maintaining data integrity of data in a storage device, the computer process comprising:
determining whether checksums stored in a checksum archive and associated with data on the storage device mounted on a computer system are trusted or untrusted, wherein the checksum archive is on a different storage device;
responsive to determining that the checksums in the checksum archive are trusted, retrieving the checksums from the checksum archive, otherwise initializing the checksums as unknown;
detecting a storage request for an identified block of the storage device;
when the storage request is a write request, calculating a checksum of the data of the identified block and updating the checksum in memory of the computer system;
when the storage request is a read request and an in-memory checksum associated with the identified block is not unknown, i) calculating the checksum of the data in the identified block and comparing the calculated checksum with an in-memory checksum associated with the identified block, and ii) determining that a data corruption has occurred when the calculated checksum does not match the in-memory checksum; and
saving checksums associated with each of the data blocks of the storage device where the data corruption has not occurred to the checksum archive.