Method and apparatus for efficiently creating backup files with less redundancy
View Patent ↗A method of creating backup files having less redundancy. The method creates a backup file by creating an overhead segment for each file that is to be backed up and creating a data segment containing the data that is to be backed up for each file. After creating the overhead segment and the data segment, the overhead segment is placed into an overhead stream data segment is stored in memory. The overhead segment is also positioned in the overhead stream with a pointer that identifies the data segment within the memory. For backups of subsequent servers or the same server at a later time, the backup software will create a separate overhead stream. However, a plurality of overhead streams may contain pointers to the same data segments such that redundant data segments do not need to be stored in a backup server.
1. A method of creating backup files, comprising:
for each file to be backed up at a server, creating a data segment, and placing the data segment into a data stream;
creating an overhead stream having a pointer that identifies the location of the data segment within the data stream;
determining if the data segment is redundant to a previously stored data segment;
if the data segment is redundant, deleting the redundant data segment and updating the overhead stream pointer to identify the previously stored data segment;
if the data segment is not redundant, storing the data segment in memory.
2. The method of claim 1 wherein the determining step comprises:
computing a checksum that represents a contents of the data segment;
comparing the checksum to previously generated checksums; and
if a matching checksum is found, identifying the data segment as redundant.
3. The method of claim 2 wherein the checksum is an MD5 checksum.
4. The method of claim 2 wherein the checksum is used to define a directory path and a file name for the data segment.
5. The method of claim 2 further comprising, for each redundant data segment, performing a binary compare between the data segment and the previously stored data segment.
6. The method of claim 2 wherein the checksum is a unique signature or fingerprint representing the contents of the data segment.
7. The method of claim 1 further comprising:
for each file to be backed up at another server, creating a data segment, and placing the data segment into another data stream;
creating another overhead stream having a pointer that identifies the location of the data segment within the other data stream.
8. The method of claim 7 further comprising:
determining if the data segment of the other data stream is redundant;
if the data segment of the other data stream is redundant, deleting the redundant data segment and updating the overhead stream pointer in the other overhead stream;
if the data segment is not redundant, storing the data segment of the other data stream in memory.
9. A non-transitory computer readable storage medium for storing software, wherein a computer implements a method in response to executing the software, the method comprising:
receiving a data segment stream from a first server, wherein the data segment stream comprises a data segment, which comprises a copy of a data file;
receiving an overhead stream from the server, wherein the overhead data stream comprises having a pointer that identifies a location of the data segment in the data stream;
determining if the data segment is redundant to a previously stored data segment;
if the data segment is redundant, deleting the redundant data segment; and
updating the overhead stream pointer to identify the previously stored data segment that is identical to the deleted data segment;
if the data segment is not redundant, storing the data segment in memory.
10. The method of claim 9 wherein the determining comprises:
computing a checksum that represents a contents of the data segment;
comparing the checksum to at least one previously generated checksums; and
if a matching checksum is found, identifying the data segment as redundant.
11. The method of claim 10 wherein the checksum is an MD5 checksum.
12. The method of claim 10 wherein the checksum is used to define a directory path and a file name for the data segment.
13. The method of claim 12 further comprising, for each redundant data segment, performing a binary compare between the data segment and the previously stored data segment.
14. The method of claim 13 wherein the checksum is a unique signature or fingerprint representing the contents of the data segment.