Machine learning engine using a distributed predictive analytics data set
View Patent ↗A novel distributed method for machine learning is described, where the algorithm operates on a plurality of data silos, such that the privacy of the data in each silo is maintained. In some embodiments, the attributes of the data and the features themselves are kept private within the data silos. The method includes a distributed learning algorithm whereby a plurality of data spaces are co-populated with artificial, evenly distributed data, and then the data spaces are carved into smaller portions whereupon the number of real and artificial data points are compared. Through an iterative process, clusters having less than evenly distributed real data are discarded. A plurality of final quality control measurements are used to merge clusters that are too similar to be meaningful. These distributed quality control measures are then combined from each of the data silos to derive an overall quality control metric.
1. A distributed method for creating a machine learning rule set, the method comprising:
preparing, on a computer, a set of data identifiers to identify data elements representing similar events for training the machine learning rule set;
sending the set of data identifiers to a plurality of data silos;
receiving a quality control metric from each data silo, wherein the quality control metric from each data silo represents the quality control metric calculated using a silo specific rule set that was derived from a machine learning algorithm using the data elements and the data identifiers on the data silo; and
combining the quality control metrics from each data silo into a combined quality control metric.
2. The method of claim 1 wherein the quality control metric is an F-Score.
3. The method of claim 1 wherein the combined quality control metric uses a weighted algorithm.
4. The method of claim 1 further comprising receiving the silo specific rule sets from at least one of the plurality of data silos.
5. The method of claim 4 further comprising receiving a plurality of silo specific rule sets and quality control metrics associated with the silo specific rule sets, from at least one of the plurality of data silos.
6. The method of claim 1 wherein the silo specific rule sets are not returned to the computer.
7. The method of claim 1 wherein a set of training results are sent with the identifiers to the plurality of data silos.
8. The method of claim 1 wherein the machine learning algorithm creates a test rule by adding a condition, calculating a test quality metric, and saving the test rule and the test quality metric if the quality metric is better than previously saved test quality metrics.
9. The method of claim 8 wherein the condition is a range locating clusters of data.
10. A non-transitory computer readable media programmed to:
prepare, on a computer, a set of data identifiers to identify data elements representing similar events for training a machine learning rule set;
send the set of data identifiers to a plurality of data silos;
receive a quality control metric from each data silo, wherein the quality control metric from each data silo represents the quality control metric calculated using a silo specific rule set that was derived from a machine learning algorithm using the data elements and the data identifiers on the data silo; and
combine the quality control metrics from each data silo into a combined quality control metric.
11. The non-transitory computer readable media of claim 10 wherein the quality control metric is an F-Score.
12. The non-transitory computer readable media of claim 10 wherein the combined quality control metric uses a weighted algorithm.
13. The non-transitory computer readable media of claim 10 wherein the silo specific rule sets are returned to the computer and combined into the machine learning rule set.
14. The non-transitory computer readable media of claim 13 wherein a plurality of silo specific rule sets and quality control metrics associated with the silo specific rule sets are returned to the computer from each data silo.
15. The non-transitory computer readable media of claim 10 wherein the silo specific rule sets are not returned to the computer.
16. The non-transitory computer readable media of claim 10 wherein an associated set of training results are sent with the identifiers to the plurality of data silos from the computer.
17. The non-transitory computer readable media of claim 10 wherein the machine learning algorithm creates a test rule by adding a condition, calculating a test quality metric, and saving the test rule and the test quality metric if the quality metric is better than previously saved test quality metrics.