Methods and apparatus for reducing storage size
Prediction-based compression engines are spoon-fed with sequentially efficiently compressible (SEC) streams of input data that make it possible for the compression engines to more efficiently compress or otherwise compact the incoming data than would be possible with streams of input data accepted on a TV-raster scan basis. Various techniques are disclosed for intentionally forming SEC input data streams. Among these are the tight packing of alike files or fragments into concatenation suitcases and the decomposition of files into substantially predictably consistent (SPC) fragments or segments that are routed to different suitcases according to their type. In a graphics-directed embodiment, image frames are partitioned into segment areas that are internally SPC and multidirectional walks (i.e., U-turning walks) are defined in the segment areas where these defined walks are traced during compression and also during decompression. A variety of pre-compression data transformation methods are disclosed for causing apparently random data sequences to appear more compressibly alike to each other. The methods are usable in systems that permit substantially longer times for data compaction operations than for data decompaction operations.
1. A machine-implemented method of classifying files for purpose of including the files or fragments thereof in a concatenation suitcase whose contents are to be supplied to one or more prediction-based compression engines, the method comprising:
(a) within a processor, scanning through a storage space of an associated memory and identifying files that are likely to be alike files by means of automatically detecting those of the potentially alike files which have one or more of:
(a.1) similar file sizes;
(a.2) similar registry extensions;
(a.3) similar other directory attributes; and
(a.4) residence within a common, high level folder that includes lower level folders;
(b) scanning through the storage space and statistically analyzing frequency of symbol occurrences in the scanned files;
(c) defining corresponding symbol occurrence histograms for the analyzed files; and
(c) identifying among the defined symbol occurrence histograms those that are most alike.
2. The classifying method of claim 1 and further comprising:
defining an automated match condition that looks for certain file signature values that indicate a high probability that condition matching files are compressibly alike to one another where the automated match condition determines at least one of the following:
if a system-provided filename of each scanned file contains a predefined character string;
if a system-provided pathname of each scanned file contains a predefined character string;
if a file header of each scanned file contains a predefined file header attribute; and
if a system-provided file size of each scanned file falls within a predefined range.
3. A machine-implemented method for classifying a given file or fragment of unknown compression type, the method comprising:
(a) within a processor, measuring or noting initial sizes of deflated ones of two or more pre-created sampler suitcases where each pre-created suitcase contains a plurality of compressibly alike files or file fragments and where each sampler suitcase has a different compression type associated with it;
(b) if not yet available in inflated form, forming an inflated version of the sampler suitcases;
(c) concatenating a copy of the given file or fragment of unknown compression type to the back of each of the two or more inflated sampler suitcases;
(d) deflating the appended-to sampler suitcases;
(e) measuring or noting the post-append sizes of the deflated two or more sampler suitcases;
(f) identifying a one of the post-append deflated sampler suitcases whose size increased least; and
(g) assigning the given file or given fragment of unknown type to the compression type of the identified sampler suitcase.
4. The classifying method of claim 3 and further comprising:
(h) defining an automated match condition that looks for certain file signature values that indicate a high probability that condition matching files are of a same compressibility type as the given file or given fragment of originally unknown type, where the automated match condition is a function of at least one of:
(h.1) a character string present in the filename of the given file or given fragment of originally unknown type;
(h.2) a character string present in the pathname of the given file or given fragment of originally unknown type; and
(h.3) a file header attribute present in a file header of the given file of originally unknown type or present in a file header containing said given fragment of originally unknown type.