System and method for self-recovering real-time data-feed compression and archiving
View Patent ↗A system and method for real-time compression and archiving of data files is disclosed. Some data files become so large that archiving onto a single storage medium, such as a CD-ROM or floppy disk, becomes difficult. In order to allow for such archival storage, an initial data file is broken down into smaller data files for compression. Once compressed, successive files are selected for storage onto a first storage medium until the storage medium becomes full. Once the first storage medium is full, the next set of data files is selected and placed onto a second storage medium. This process is repeated until all of the smaller files that originated from the initial file are archived.
1. A method of archiving data files onto a storage medium having a fixed storage capacity, comprising:
receiving a data file;
dividing said data file into a plurality of divisional files;
separately compressing each of the plurality of divisional files to obtain compressed files;
sequentially adding successive compressed files to a directory;
checking a size of the directory each time a new compressed file is to be added to ensure that adding the new compressed file does not exceed the storage capacity of the fixed medium;
if adding the new compressed file does not cause a group size to exceed the storage capacity of the fixed medium, adding the new compressed file to the directory; and
if adding the new file would cause the group size to exceed the storage capacity of the fixed medium, transferring the compressed files from the directory to the storage medium.
2. The method of claim 1 , wherein receiving a data file comprises receiving a log file.
3. The method of claim 1 , wherein said dividing said data file into one or more divisional files comprises dividing said data file into one or more divisional files based on time intervals.
4. The method of claim 3 , wherein compressing the one or more divisional files comprises zipping the one or more divisional files.
5. The method of claim 1 , wherein said transferring said group to the storage medium comprises transferring said group to the storage medium via a wired communications link.
6. The method of claim 1 , wherein said transferring said group to the storage medium comprises transferring said group to the storage medium via a wireless communications link.
7. The method of claim 1 , wherein the adding, deleting, and transferring are repeated until all of the plurality of compressed files have been stored.
8. The method of claim 1 , further comprising updating a place marker indicative of a last successful archiving step.
9. The method of claim 8 , further comprising detecting a system fault and accessing the place marker.
10. The method of claim 9 , further comprising resuming archiving at a stage indicated by the place marker.
11. The method of claim 1 , wherein the storage medium comprises a floppy disk.
12. The method of claim 1 , wherein the storage medium comprises a hard disk.
13. The method of claim 1 , wherein the storage medium comprises a magnetic tape.
14. The method of claim 1 , wherein the storage medium comprises an optical disk.
15. A system of archiving data files onto a storage medium, said system comprising:
a storage medium;
a processing unit;
a program module operable on said processing unit, said program module containing instructions operative to:
receive a data file;
divide said data file into one or more divisional files;
compress said one or more divisional files;
sequentially add successive compressed files to a directory;
check a size of the directory each time a new compressed file is to be added to ensure that adding the new compressed file does not exceed the capacity of the storage medium;
collect one or more compressed files into a group approximately the size of but not exceeding the size of the storage medium; and
transfer said group to the storage medium.
16. The system of claim 15 , wherein said storage medium comprises a floppy disk.
17. The system of claim 15 , wherein said storage medium comprises a hard disk.
18. The system of claim 15 , wherein said storage medium comprises magnetic tape.
19. The system of claim 15 , wherein the storage medium comprises an optical disk.