SYSTEMS AND METHODS FOR ELECTRONIC DATA CLUSTER ANALYSIS
A computer-implemented method for electronic data cluster analysis may include receiving a plurality of electronic data files including non-human-readable data, calculating a similarity of each pair of electronic data files among the plurality of electronic data files, determining one or more electronic data file clusters among the plurality of electronic data files based on the calculated similarity of each pair of electronic data files among the plurality of electronic data files, and extracting one or more commonalities in each electronic data file cluster among the determined one or more electronic data file clusters
1 . A computer-implemented method for electronic data cluster analysis, the method comprising:
receiving a plurality of electronic data files including non-human-readable data;
calculating a similarity of each pair of electronic data files among the plurality of electronic data files;
determining one or more electronic data file clusters among the plurality of electronic data files based on the calculated similarity of each pair of electronic data files among the plurality of electronic data files; and
extracting one or more commonalities in each electronic data file cluster among the determined one or more electronic data file clusters.
2 . The computer-implemented method of claim 1 , further comprising:
reporting the extracted one or more commonalities.
3 . The computer-implemented method of claim 1 , further comprising:
preprocessing the non-human-readable data for each electronic data file among the plurality of electronic data files,
wherein preprocessing the non-human-readable data for each electronic data file comprises:
constructing a list of common strings from the non-human-readable data for each electronic data file.
4 . The computer-implemented method of claim 1 , wherein calculating a similarity of each pair of electronic data files comprises:
counting instances of each string of the common strings appearing in the non-human-readable data for each electronic data file; and
calculating the similarity of each pair of electronic data files based on the counted instances.
5 . The computer-implemented method of claim 4 , wherein the similarity of each pair of electronic data files is calculated as a positive number.
6 . The computer-implemented method of claim 1 , wherein determining one or more electronic data file clusters comprises iteratively performing until a desired number of electronic data file clusters is determined:
searching the calculated similarities of each pair of electronic data file for a maximum similarity between pairs of electronic data files;
averaging the similarities of the pair of electronic data files having the maximum similarity into a new electronic data file cluster;
remove individual electronic data files of the pair of electronic data files having the maximum similarity from the plurality of electronic data files;
adding the new electronic data file cluster to the plurality of electronic data files; and
calculating a similarity of the new cluster and each electronic data file among the plurality of electronic data files.
7 . The computer-implemented method of claim 1 , wherein the non-human-readable data is encoded data.
8 . The computer-implemented method of claim 1 , wherein extracting one or more commonalities in each electronic data file cluster further comprises:
counting instances of each string appearing in the non-human-readable data for each electronic data file and each cluster; and
calculating the similarity of each electronic data file and each cluster based on the counted instances of each string.
9 . The computer-implemented method of claim 1 , wherein a number of extracted commonalities in each electronic data file cluster is a user-specified maximum number of commonalities.
10 . The computer-implemented method of claim 1 , wherein a number of determined clusters is a user-specified maximum number of clusters.
11 . The computer-implemented method of claim 1 , further comprising:
identifying one or more outlier electronic data files among the plurality of electronic data files and one or more commonalities in the one or more outlier electronic data files.
12 . A system for electronic data cluster analysis, the system comprising:
at least one data storage device storing instructions for electronic data cluster analysis in an electronic storage medium; and
at least one processor configured to execute the instructions to perform operations including:
receiving a plurality of electronic data files including non-human-readable data;
preprocessing the non-human-readable data for each electronic data file among the plurality of electronic data files;
calculating a similarity of each pair of electronic data files among the plurality of electronic data files;
determining one or more electronic data file clusters among the plurality of electronic data files based on the calculated similarity of each pair of electronic data files among the plurality of electronic data files; and
extracting one or more commonalities in each electronic data file cluster among the determined one or more electronic data file clusters.
13 . The system of claim 12 , wherein preprocessing the non-human-readable data for each electronic data file comprises:
constructing a list of common strings from the non-human-readable data for each electronic data file.
14 . The system of claim 12 , wherein calculating a similarity of each pair of electronic data files comprises:
counting instances of each string of the common strings appearing in the non-human-readable data for each electronic data file; and
calculating the similarity of each pair of electronic data files based on the counted instances.
15 . The system of claim 12 , wherein determining one or more electronic data file clusters comprises iteratively performing until a desired number of electronic data file clusters is determined:
searching the calculated similarities of each pair of electronic data file for a maximum similarity between pairs of electronic data files;
averaging the similarities of the pair of electronic data files having the maximum similarity into a new electronic data file cluster;
remove individual electronic data files of the pair of electronic data files having the maximum similarity from the plurality of electronic data files;
adding the new electronic data file cluster to the plurality of electronic data files; and
calculating a similarity of the new cluster and each electronic data file among the plurality of electronic data files.
16 . The system of claim 12 , wherein extracting one or more commonalities in each electronic data file cluster further comprises:
counting instances of each string appearing in the non-human-readable data for each electronic data file and each cluster; and
calculating the similarity of each electronic data file and each cluster based on the counted instances of each string.
17 . A non-transitory machine-readable medium storing instructions that, when executed by a computing system, causes the computing system to perform operations for electronic data cluster analysis, the operations comprising:
receiving a plurality of electronic data files including non-human-readable data;
preprocessing the non-human-readable data for each electronic data file among the plurality of electronic data files;
calculating a similarity of each pair of electronic data files among the plurality of electronic data files;
determining one or more electronic data file clusters among the plurality of electronic data files based on the calculated similarity of each pair of electronic data files among the plurality of electronic data files; and
extracting one or more commonalities in each electronic data file cluster among the determined one or more electronic data file clusters.
18 . The non-transitory machine-readable medium of claim 17 , wherein preprocessing the non-human-readable data for each electronic data file comprises:
constructing a list of common strings from the non-human-readable data for each electronic data file.
19 . The non-transitory machine-readable medium of claim 17 , wherein determining one or more electronic data file clusters comprises iteratively performing until a desired number of electronic data file clusters is determined:
searching the calculated similarities of each pair of electronic data file for a maximum similarity between pairs of electronic data files;
averaging the similarities of the pair of electronic data files having the maximum similarity into a new electronic data file cluster;
remove individual electronic data files of the pair of electronic data files having the maximum similarity from the plurality of electronic data files;
adding the new electronic data file cluster to the plurality of electronic data files; and
calculating a similarity of the new cluster and each electronic data file among the plurality of electronic data files.
20 . The non-transitory machine-readable medium of claim 17 , wherein extracting one or more commonalities in each electronic data file cluster further comprises:
counting instances of each string appearing in the non-human-readable data for each electronic data file and each cluster; and
calculating the similarity of each electronic data file and each cluster based on the counted instances of each string.