Generating and merging keys for grouping and differentiating volumes of files
View Patent ↗Methods and apparatus teach a digital spectrum of a file. The digital spectrum is used to map a file's position in a multi-dimensional space. This position relative to another file's position reveals distances between the files. Closest files can be grouped together. When contemplating voluminous numbers of files for digital spectrums, various methods include: concatenating all such files together to get a single key useful for creating a file's spectrum; or compressing files individually and combining their collective dictionaries into a single dictionary with or without the use of tree mechanisms that defines the digital spectrum. Each provides advantage over the other. The latter consumes considerably less run time because each compression event can be distributed to a separate processor. Method two provides better spectrums because it is more “informationally” valid than is method one.
1. In a computing system environment, a method of differentiating a plurality of files stored on one or more computing devices, comprising:
compressing each file from the plurality of files, and during the compressing creating a plurality of individual keys one key for each of the plurality of files and each key representing symbol patterns for information content in the file to which it relates; and
converting and merging the plurality of individual keys into a single set key that reflects all the symbol patterns occurring over the files, and each individual key is processed for deriving a specific digital spectrum for a given file for which it represents and each individual key compared to remaining keys for providing a relationship for the file to which that individual key relates relative to each of remaining ones of the files and those remaining files' keys, and wherein each key for each file is an algorithmic sum of other smaller keys associated with that file, and wherein each key is summed to include the other smaller keys as addends within that key.
2. The method of claim 1 , further including using the single set key to create a digital spectrum for each of the plurality of files.
3. The method of claim 2 , further including using the created digital spectrum to find closest files in a multi-dimensional space.
4. The method of claim 3 , wherein the finding closest files in the multi-dimensional space further includes computing a distance value between each file and every other file of the plurality of files.
5. The method of claim 1 , wherein the converting includes reading a dictionary portion of a first of the plurality of individual keys.
6. The method of claim 5 , wherein the reading the dictionary portion of the first of the plurality of individual keys includes storing each of a plurality of first symbols in the dictionary portion in terms of a first pair of symbol ID's.
7. The method of claim 6 , wherein the converting further includes reading a subsequent dictionary portion corresponding to each subsequent individual key of the plurality of the individual keys.
8. The method of claim 7 , wherein the reading the subsequent dictionary portion corresponding to each subsequent individual key of the plurality of individual keys includes storing each of a plurality of second symbols in the subsequent dictionary portion in terms of a second pair of symbol ID's.
9. The method of claim 8 , further including comparing the first pair of symbol ID's with the second pair of symbol ID's for each of the plurality of first and second symbols.
10. The method of claim 9 , wherein the comparing includes updating a translation table when any one of the first pair of symbol ID's match any of the second pair of symbol ID's to reflect that that a new symbol II) should be translated to a previously defined symbol II).
11. The method of claim 9 , wherein the comparing includes updating a translation table when any one of the first pair of symbol ID's does not match any of the second pair of symbol ID's such that a new symbol ID is defined in the translation table.
12. In a computing system environment, a method of differentiating a plurality of files stored on one or more computing devices, each file having a plurality of symbols representing an underlying data stream of original bits of data, comprising:
compressing each file from the plurality of files, and during the compressing generating a plurality of individual keys one key for each said plurality of files and each key representing symbol patterns for information content in the file to which it relates, wherein each individual key for each file is an algorithmic sum of other smaller keys for that file, and wherein each key is summed to include the other smaller keys as addends within that key;
converting and merging the plurality of individual keys into a single set key that reflects all the symbol patterns occurring over the files, and each individual key is processed for deriving a specific digital spectrum for a given file for which it represents, and each individual key compared to remaining keys for providing a relationship for the file to which that individual key relates relative to each of remaining ones of the files and those remaining files' keys; and
generating a digital spectrum in a multi-dimensional space using only the single set key.
13. The method of claim 12 , further including using the generated digital spectrum to find closest other files in the multi-dimensional space.
14. The method of claim 12 , further including distributing a compression task to a plurality of processors on one or more computing devices so that the compressing individually each file of the plurality of files occurs in parallel for each file thereby minimizing an overall compression runtime.
15. The method of claim 12 , further including performing the compressing sequentially for each of the plurality of files on a single processor on a single computing device.
16. The method of claim 12 , wherein the compressing further includes providing each of the plurality of individual keys with a corresponding one of a plurality of dictionaries having a plurality of symbol entries.
17. The method of claim 16 , wherein the converting further includes providing the single set key with fewer symbol entries in a combined dictionary than a combined number of the plurality of symbol entries in the plurality of dictionaries for the plurality of individual keys.
18. In a computing system environment, a method of merging a plurality of individual keys each corresponding to a plurality of files stored into one set key, comprising:
compressing each file from the plurality of files, and during the compressing creating the plurality of individual keys each key corresponding to one file of the plurality of files and each key representing symbol patterns for information content in the file to which it relates, and each individual key for each file is an algorithmic sum of other smaller keys for that file, and wherein each key is summed to include the other smaller keys as addends within that key;
summing and merging each of the plurality of individual keys for each file of the plurality of files into the single set key having fewer symbol entries than a combined number of symbol entries for the plurality of individual keys, wherein the single set key reflects all the symbol patterns occurring over the files representing information content for the plurality of files, and each individual key is processed for deriving a specific digital spectrum for a given file for which it represents, and each individual key is compared to remaining keys for providing a relationship for the file to which that individual key relates relative to each of remaining ones of the files and those remaining files' keys; and
using the set key to create a digital spectrum for each of the plurality of files.
19. The method of claim 18 , wherein the summing includes comparing a plurality of underlying bit patterns of each of the symbol entries for each of the plurality of individual keys.
20. The method of claim 19 , wherein the summing further includes updating a translation table when any of the plurality of underlying bit patterns of each of the symbol entries for each of the plurality of individual keys match such that extraneous data is eliminated.