Grouping and differentiating volumes of files
View Patent ↗Methods and apparatus teach a digital spectrum of a file. The digital spectrum is used to map a file's position in a multi-dimensional space. This position relative to another file's position reveals distances between the files. Closest files can be grouped together. When contemplating voluminous numbers of files for digital spectrums, various methods include: concatenating all such files together to get a single key useful for creating a file's spectrum; or compressing files individually and combining their collective dictionaries into a single dictionary that defines the digital spectrum. Each provides advantage over the other. The latter consumes considerably less run time because each compression event can be distributed to a separate processor. Method two provides better spectrums because it is more “informationally” valid than is method one.
1. In a computing system environment, a method of differentiating pluralities of files stored on one or more computing devices, each file having a plurality of symbols representing an underlying data stream of original bits of data, comprising:
compressing individually said each file of the plurality of files, the compressing generating a plurality of dictionaries one each for said each file, the compressing achieved by identifying all possible tuples in the files for sets of characters and replacing each unique tuple with a particular symbol and each particular symbol stored in the single dictionary, and generating a particular new symbol for a most frequently identified tuple and replacing it in the single dictionary and when multiple tuples have a same frequency as the most frequently identified tuple, resolving the most frequency identified tuple by projecting each of the multiple tuples into right triangles and solving for hypotenuses, a particular hypotenuse having a longest length from remaining hypotenuses being identified as the most frequently identified tuple;
converting the pluralities of dictionaries into the single dictionary having fewer entries than a combined number of entries for the plurality of dictionaries; and
generating a digital spectrum in a multi-dimensional space using only the single dictionary.
2. The method of claim 1 , further including distributing a compression task to a plurality of processors on one or more computing devices so that the compressing individually said each file of the plurality of files occurs in parallel for said each file thereby minimizing an overall compression runtime.
3. The method of claim 1 , further including using the generated digital spectrum to find closest other files in the multi-dimensional space.
4. The method of claim 1 , wherein the converting the pluralities of dictionaries into the single dictionary further includes adding entries to a first of the plurality of dictionaries from a second of the plurality of dictionaries.
5. The method of claim 4 , wherein the adding entries further includes only adding new entries for original entries in the second of the plurality of dictionaries not already included in the first of the plurality of dictionaries.
6. The method of claim 1 , furthering including determining an encoding tree for a first of the plurality of dictionaries.
7. The method of claim 6 , further including adding non-redundant entries to the encoding tree from a second of the plurality of dictionaries.