Synchronization of storage using comparisons of fingerprints of blocks
A system for processing data includes a data storage device and a processor. The data storage device stores a set of data. The processor is configured to divide the set of data in the data storage system into a set of segments; compute a set of fingerprints, wherein the set of fingerprints comprises a fingerprint for each segment of the set of segments; store the set of fingerprints in a new snapshot; identify a second set of fingerprints in the new snapshot that are not already in a fingerprint index; cause a second set of segments associated with the second set of fingerprints to be stored in a backup data storage system; and cause the second set of fingerprints to be added to the fingerprint index.
1. A system for processing data comprising:
a data storage device, wherein the data storage device stores a set of data; and
a processor is configured to:
divide the set of data in the data storage device into a set of segments;
compute a set of fingerprints, wherein the set of fingerprints comprises a fingerprint for each segment of the set of segments;
store the set of fingerprints in a new snapshot stored in the data storage device, wherein the new snapshot comprises an indication of the set of segments at a given point in time;
identify a second set of fingerprints in the new snapshot that are not already in a fingerprint index including comparing each fingerprint in the new snapshot with each fingerprint in the fingerprint index;
cause a second set of segments associated with the second set of fingerprints to be stored in a backup data storage system;
cause the second set of fingerprints to be added to the fingerprint index;
identify a third set of fingerprints in the new snapshot that are not found in a list of fingerprints in a target snapshot;
delete a first subset of data stored in the data storage device, wherein the first subset of data are associated with the third set of fingerprints;
retrieve a second subset of data from the backup data storage system, wherein the second subset of data are associated with fingerprints that are found in the list of fingerprints in the target snapshot; and
insert the second subset of data into appropriate locations in the data storage device.
2. The system as in claim 1 , wherein the data storage device for storing data comprises a block storage system.
3. The system as in claim 1 , wherein the backup data storage system comprises a deduplicating storage system.
4. The system as in claim 1 , wherein a block is associated with one or more segments of the set of segments.
5. The system as in claim 4 , wherein the block is restored by reading the one or more segments from the backup data storage system and used to reconstruct the block.
6. The system as in claim 5 , wherein the processor is further configured to transfer the one or more segments from the backup data storage system to the data storage device.
7. The system as in claim 6 , wherein transferring the one or more segments comprises transferring the one or more segments requested by a user.
8. The system as in claim 1 , wherein the new snapshot is one of a plurality of snapshots.
9. The system as in claim 1 , wherein the new snapshot is associated with a snapshot time.
10. A method for processing data, comprising:
dividing a set of data in a data storage device into a set of segments;
computing, using a processor, a set of fingerprints, wherein the set of fingerprints comprises a fingerprint for each segment of the set of segments;
storing the set of fingerprints in a new snapshot stored in the data storage device, wherein the new snapshot comprises an indication of the set of segments at a given point in time;
identifying a second set of fingerprints in the new snapshot that are not already in a fingerprint index including comparing each fingerprint in the new snapshot with each fingerprint in the fingerprint index;
causing a second set of segments associated with the second set of fingerprints to be stored in a backup data storage system;
causing the second set of fingerprints to be added to the fingerprint index;
identifying a third set of fingerprints in the new snapshot that are not found in a list of fingerprints in a target snapshot;
deleting a first subset of data stored in the data storage device, wherein the first subset of data are associated with the third set of fingerprints;
retrieving a second subset of data from the backup data storage system, wherein the second subset of data are associated with fingerprints that are found in the list of fingerprints in the target snapshot; and
inserting the second subset of data into appropriate locations in the data storage device.
11. The method as in claim 10 , wherein the data storage device for storing data comprises a block storage system.
12. The method as in claim 10 , wherein the backup data storage system comprises a deduplicating storage system.
13. The method as in claim 10 , wherein a block is associated with one or more segments of the set of segments.
14. The method as in claim 13 , wherein the block is restored by reading the one or more segments from the backup data storage system and used to reconstruct the block.
15. The method as in claim 14 , further comprising transferring the one or more segments from the backup data storage system to the data storage device.
16. The method as in claim 15 , wherein transferring the one or more segments comprises transferring the one or more segments requested by a user.
17. The method as in claim 10 , wherein the new snapshot is one of a plurality of snapshots.
18. The method as in claim 10 , wherein the new snapshot is associated with a snapshot time.
19. A computer program product for processing data, the computer program product being embodied in a non-transitory computer readable storage medium and comprising computer instructions for:
dividing a set of data in a data storage device into a set of segments;
computing, using a processor, a set of fingerprints, wherein the set of fingerprints comprises a fingerprint for each segment of the set of segments;
storing the set of fingerprints in a new snapshot stored in the data storage device, wherein the new snapshot comprises an indication of the set of segments at a given point in time;
identifying a second set of fingerprints in the new snapshot that are not already in a fingerprint index including comparing each fingerprint in the new snapshot with each fingerprint in the fingerprint index;
causing a second set of segments associated with the second set of fingerprints to be stored in a backup data storage system;
causing the second set of fingerprints to be added to the fingerprint index;
identifying a third set of fingerprints in the new snapshot that are not found in a list of fingerprints in a target snapshot;
deleting a first subset of data stored in the data storage device, wherein the first subset of data are associated with the third set of fingerprints;
retrieving a second subset of data from the backup data storage system, wherein the second subset of data are associated with fingerprints that are found in the list of fingerprints in the target snapshot; and
inserting the second subset of data into appropriate locations in the data storage device.