System and method for filesystem data compression using codebooks
A system and method for filesystem data compression using codebooks, that measures in real-time the probability distribution of an encoded data stream, compares the probability distribution to a reference probability distribution, and uses one or more statistical algorithms to determine the divergence between the two sets of probability distributions to determine if an unusual distribution is the result of a data intrusion. The system comprises both encoding and decoding machines, an intrusion detection module, a codebook training module, and various databases which perform various analyses on encoded data streams. Further, the system comprises a system for integrating the compression into a filesystem for both system-wide compression on a per-file or filegroup basis, and intrusion or alteration detection of files.
1 . A system for file system data compaction using codebooks, comprising:
a computing device comprising a processor, a memory, and a persistent file system;
a codebook library manager comprising a plurality of programming instructions that, when operating on the processor, cause the processor to:
receive a digital file to compact;
analyze the digital file to dynamically determine its statistical properties;
generate a file-specific codebook for compacting the digital file based on the statistical properties, wherein the file-specific codebook is generated from statistical analysis of the content within the individual digital file being compacted, and is specific to that individual file;
compact the digital file using the generated file-specific codebook;
save both the generated file-specific codebook and the compacted digital file on the computing device to the persistent file system;
wherein the compacted digital file includes a persistent reference or link to the generated file-specific codebook in the persistent file system of the computing device; and
wherein an inability to perfectly decode the compacted digital file using the linked codebook when access is requested from the persistent file system indicates tampering with the digital file.
2 . The system of claim 1 , wherein the codebook library manager and the target file system are operating on separate computing devices.
3 . The system of claim 2 , wherein the separate computing devices communicate over a network.
4 . The system of claim 1 , wherein the compacted digital file is encrypted.
5 . The system of claim 1 , wherein the process of compacting the digital file applies an encryption algorithm to the compaction process.
6 . A method for file system data compaction using codebooks, comprising the steps of:
receiving digital file to compact;
analyzing the digital file to dynamically determine its statistical properties; and
generating a file-specific codebook for compacting the digital file based on the statistical properties, wherein the file-specific codebook is generated from statistical analysis of the content within the individual digital file being compacted, and is specific to that individual file;
compacting the digital file using the generated file-specific codebook;
saving both the generated file-specific codebook and the compacted digital file on the computing device to a persistent file system configured for long-term storage of digital files;
wherein the compacted digital file includes a persistent reference or link to the generated file-specific codebook in the persistent file system; and
wherein an inability to perfectly decode the compacted digital file using the linked codebook when access is requested from the persistent file system indicates tampering with the digital file.
7 . The method of claim 6 , wherein the codebook library manager and the target filesystem are operating on separate computing devices.
8 . The method of claim 7 , wherein the separate computing devices communicate over a network.
9 . The method of claim 6 , wherein the compacted digital file is encrypted.
10 . The method of claim 6 , wherein the process of compacting the digital file applies an encryption algorithm to the compaction process.