Hybrid clustering for data analytics
The invention includes methods and systems for analyzing data to determine trends in the data and to identify outliers. The methods and systems include a learning algorithm whereby a data space is co-populated with artificial, evenly distributed data, and then the data space is carved into smaller portions whereupon the number of real and artificial data points are compared. Through an iterative process, clusters having less than evenly distributed real data are discarded. Additionally, a final quality control measurement is used to merge clusters that are too similar to be meaningful. The invention is widely applicable to data analytics, generally, including financial transactions, retail sales, elections, and sports.
1. A computer implemented method for clustering data, the method comprising:
(a) obtaining real data from a computer readable medium;
(b) identifying a rule that clusters the real data, wherein identifying comprises:
i) generating an evenly-distributed set of synthesized data having an identical number of points as the real data;
ii) creating a database on the computer readable medium including the real data and the synthesized data;
iii) executing a supervised machine-learning algorithm on a processor to determine a rule for clustering the real data, such that the real data is differentiated from the synthesized data:
iv) clustering the real data using the determined rule;
v) evaluating the quality of the determined rule for clustering the real data by:
calculating a first quality statistic based on a percentage of the real data clustered by the rule; and
calculating a second quality statistic based on a percentage of the real data and the synthesized data covered by the rule that is synthesized data;
wherein a rule resulting in a larger first quality statistic and a smaller second quality statistic is evaluated more favorably than a rule resulting in a smaller first quality statistic and a larger second quality statistic;
vi) correlating the clustered real data with a data space using a processor; and
vii) validating the rule by comparing the location of the clustered real data with a location of other clustered real data using a processor;
(c) clustering the real data with the rule using a processor; and
(d) storing the clustered real data on the computer readable medium.
2. The method of claim 1 , wherein validating comprises calculating a centroid of the clustered real data using a processor and calculating a distance between the centroid of the clustered real data and the centroid of a nearest neighboring cluster formed by the same rule.
3. The method of claim 2 , wherein validating further comprises calculating an average distance between each real data point in the cluster and the centroid of the clustered real data.
4. The method of claim 3 , further comprising calculating a ratio of (the average distance between each real data point in the cluster and the centroid) to (the distance between the centroid and the centroid of the nearest neighboring cluster formed by the same rule).
5. The method of claim 1 , further comprising calculating a centroid for each cluster of real data on the computer readable medium, and comparing the distribution of centroids from the clusters of real data on the computer readable medium to determine a trend in the real data.
6. The method of claim 1 , further comprising calculating a centroid for each cluster of real data on the computer readable medium, and comparing the distribution of centroids from the clusters of real data on the computer readable medium to determine an outlier in the real data.
7. The method of claim 1 , wherein the real data relates to genetic sequences.
8. The method of claim 1 , wherein the real data relates to stock, commodity, currency, or retail sales.
9. The method of claim 1 , wherein the real data relates to athletic performance.
10. The method of claim 1 , wherein the real data is data reported to a government authority.
11. A system for clustering data comprising a processor, memory, and a computer readable medium, wherein the computer readable medium comprises instructions that when executed cause the processor to:
obtain real data from the computer readable medium;
identify a rule that clusters the real data, wherein identifying comprises:
i) generating an evenly-distributed set of synthesized data having an identical number of points as the real data;
ii) creating a database on the computer readable medium including the real data and the synthesized data;
iii) executing a supervised machine-learning algorithm to determine a rule for clustering the real data, such that the real data is differentiated from the synthesized data;
iv) clustering the real data using the determined rule;
v) evaluating the Quality of the determined rule for clustering the real data by:
calculating a first quality statistic based on a percentage of the real data clustered by the rule; and
calculating a second quality statistic based on a percentage of the real data and the synthesized data covered by the rule that is synthesized data;
vi) correlating the clustered real data with a data space; and
vii) validating the rule by comparing the location of the clustered real data with a location of other clustered real data;
cluster the real data with the rule; and
store the clustered real data on the computer readable medium.
12. The system of claim 11 , wherein validating comprises calculating a centroid of the clustered real data and calculating a distance between the centroid of the clustered real data and the centroid of a nearest neighboring cluster formed by the same rule.
13. The system of claim 12 , wherein validating further comprises calculating an average distance between each real data point in the cluster and the centroid of the clustered real data.
14. The system of claim 13 , further comprising calculating a ratio of (the average distance between each real data point in the cluster and the centroid) to (the distance between the centroid and the centroid of the nearest neighboring cluster formed by the same rule).
15. The system of claim 11 , wherein the computer readable medium further comprises instructions that cause the processor to calculate a centroid for each cluster of real data on the computer readable medium, and compare the distribution of centroids from the clusters of real data on the computer readable medium to determine a trend in the real data.
16. The system of claim 11 , wherein the computer readable medium further comprises instructions that cause the processor to calculate a centroid for each cluster of real data on the computer readable medium, and compare the distribution of centroids from the clusters of real data on the computer readable medium to determine an outlier in the real data.
17. The system of claim 11 , wherein the real data relates to genetic sequences.
18. The system of claim 11 , wherein the real data relates to stock, commodity, currency, or retail sales.
19. The system of claim 11 , wherein the real data relates to athletic performance.
20. The system of claim 11 , wherein the real data is data reported to a government authority.