Generating and merging keys for grouping and differentiating volumes of files
Methods and apparatus teach a digital spectrum of a file. The digital spectrum is used to map a file's position in a multi-dimensional space. This position relative to another file's position reveals distances between the files. Closest files can be grouped together. When contemplating voluminous numbers of files for digital spectrums, various methods include: concatenating all such files together to get a single key useful for creating a file's spectrum; or compressing files individually and combining their collective dictionaries into a single dictionary with or without the use of tree mechanisms that defines the digital spectrum. Each provides advantage over the other. The latter consumes considerably less run time because each compression event can be distributed to a separate processor. Method two provides better spectrums because it is more “informationally” valid than is method one.
1. A method, comprising:
creating, by executable instructions that execute on a hardware processor from a non-transitory computer-readable storage medium, keys for patterns present in files during compression of the files;
totaling, by the executable instructions, a total number of unique keys present in the keys for the files;
summing, by the executable instructions, each file's keys into a single file key by creating an ordered vector for each file that comprises ordered pairs of scalar values, particular scalar values in a particular ordered pair vector representing a particular one of the keys and a particular frequency count for that particular one of the keys;
mapping, by the executable instructions each file multidimensional space based on processing each file's single file key, wherein a total number of dimensions for the multidimensional space is equal to the total number of unique keys, wherein mapping further includes plotting each ordered vector represented in each single file key as a series of coordinates within the multidimensional space defined by the corresponding ordered pairs; and
identifying, by the executable instructions, content relationships between each file to remaining ones of the files based on distances between each of the files mapped in the multidimensional space.
2. The method of claim 1 , wherein creating further includes assigning key lengths for the keys based on frequencies of each pattern appearing within the files, wherein shorter key lengths are assigned to more frequently appearing patterns and longer key lengths are assigned to less frequency appearing patterns within the files.
3. The method of claim 2 , wherein creating further includes matching the patterns to symbols in a dictionary of symbols for the files during compression.
4. The method of claim 3 , wherein matching further includes replacing the patterns in the files with the symbols obtained from the dictionary during compression.
5. The method of claim 4 , wherein creating further includes distributing the files to multiple processors for parallel processing during compression.
6. The method of claim 5 , wherein identifying further includes finding for any given file a closest related file based on a computed distance between the given file and the closest related file being less than other computed distances between the given file and other remaining ones of the files.
7. The method of claim 1 , wherein mapping further includes generating a composite key that represents the single file keys.
8. The method of claim 7 , wherein generating further includes defining the composite key as a set key for the single file keys.
9. The method of claim 8 , wherein generating further includes creating a digital spectrum within the multidimensional space for each file using the set key.
10. The method of claim 8 , wherein defining further includes associating the composite key with a dictionary of symbols that has fewer symbol entries than a plurality of dictionaries associated with all of the single file keys.
11. The method of claim 10 , wherein summing further includes associating each single file key to a unique one of the plurality of dictionaries for the file that relates to that single file key.