Ranking importance of symbols in underlying grouped and differentiated files based on content
View Patent ↗Methods and apparatus identify groups of files based on symbols corresponding to an underlying data stream of original bits of data that are determined to be informationally important. The resulting symbols of a selected group are ordered according to how effectively each symbol characterizes the selected group of interest. The subset of symbols is used to find similar files from a general population of files to the files in the group of interest. Additionally, groups of common files can be identified from a general population of files and a group selected therefrom for use in identifying a subset of symbols which characterize the selected group for use as a filter to identify further like files.
1. A method for identifying a subset of symbols from a group of symbols that most effectively characterizes a group of files for a plurality of files stored on one or more computing devices, each file of said group of files having a plurality of symbols forming at least a portion of the group of symbols and representing an underlying data stream of original bits of data, the method residing in a non-transitory computer-readable storage medium as executable instructions processed by the one or more computing devices and comprising:
determining an amount of information represented by each symbol of said group of symbols by determining a per file amount of information represented by said each symbol of said group of symbols for said group of files, and summing said per file amount of information for said each symbol of said group of symbols for said group of files and ordering said group of symbols based on said summed per file amount of information and selecting a portion of said ordered symbols for use in the calculating, wherein said selected portion of said ordered symbols represents substantially between 80%-99% of a total amount of information within said group of files; and
calculating a variance for said each symbol of said group of symbols.
2. The method of claim 1 , further including ordering said group of symbols based on said amount of information before calculating the variance of said each symbol of said group of symbols.
3. The method of claim 1 , wherein said selected portion of said ordered symbols represents substantially 90% of said total amount of information within said group of files.
4. The method of claim 2 , wherein the determining said amount of information includes determining a per file amount of information represented by said each symbol of said group of symbols for said group of files by multiplying a determined number of occurrences for said each symbol of said group of symbols by a corresponding bit length of said original bits of data represented by said each symbol, and dividing by a total of said original bits of data for each file.
5. The method of claim 4 , wherein the determining further includes summing said per file amount of information for each symbol of said group of symbols for said group of files.
6. The method of claim 5 , wherein the ordering is based on said summed amount of information.
7. A method for identifying a subset of symbols from a group of symbols that most effectively characterizes a group of files for a plurality of files stored on one or more computing devices, each file of said group of files having a plurality of symbols forming at least a portion of the group of symbols and representing an underlying data stream of original bits of data, the method residing in a non-transitory computer-readable storage medium as executable instructions processed by the one or more computing devices and comprising:
determining an amount of information represented by each symbol of said group of symbols by determining a per file amount of information represented by said each symbol of said group of symbols for said group of files, and summing said per file amount of information for said each symbol of said group of symbols for said group of files and ordering said group of symbols based on said summed per file amount of information and selecting a portion of said ordered symbols for use in the calculating, wherein said selected portion of said ordered symbols represents substantially between 80%-99% of a total amount of information within said group of files;
calculating a variance for said each symbol of said group of symbols; and
ordering said group of symbols based on said amount of information before calculating the variance of said each symbol of said group of symbols, and selecting a portion of said ordered symbols and determining a mean and a standard deviation from said mean for each symbol of said portion of said ordered symbols and ordering said portion of said ordered symbols, wherein the identified subset of symbols from said group of symbols includes substantially 10% of said ordered portion of said ordered symbols.
8. The method of claim 1 , further including identifying at least one additional file of said plurality of files stored on said one or more computing devices based on said determined amount of information and said calculated variance.
9. The method of claim 2 , further including filtering said plurality of files stored on one or more computing devices utilizing said subset of symbols from said group of symbols which most effectively characterize said group of files.