Learning data processor for distributing learning machines across large-scale network infrastructures
In one embodiment, a learning data processor determines a plurality of machine learning features in a computer network to collect. Upon receiving data corresponding to the plurality of features, the learning data processor may aggregate the data, and pushes the aggregated data for select features to interested learning machines associated with the computer network.
1. A method, comprising:
determining a plurality of machine learning features in a computer network to collect at a learning data processor;
receiving data corresponding to the plurality of features;
aggregating the data;
generating a statistical analysis of all collected features;
performing a traffic reduction measure based on the statistical analysis;
learning a probabilistic model of time evolution of the received feature data; and
pushing the aggregated data for select features to interested learning machines associated with the computer network.
2. The method as in claim 1 , wherein pushing comprises:
multicasting each type of aggregated data within a corresponding multicast group, wherein interested learning machines join multicast groups for features in which they are interested.
3. The method as in claim 1 , further comprising:
receiving, from the interested learning machines, an indication of features in which they are interested.
4. The method as in claim 1 , further comprising:
establishing and using a control multicast group for the learning data processor to communicate control messages with all learning machines associated with the computer network.
5. The method as in claim 1 , further comprising:
using a control application port for the learning data processor to communicate control messages with all learning machines associated with the computer network.
6. The method as in claim 1 , further comprising:
selecting the selected features to push as only those collected features in which one or more learning machines associated with the network are interested.
7. The method as in claim 1 , wherein the traffic reduction measure comprises:
combining correlated features into a representative feature.
8. The method as in claim 1 , wherein the traffic reduction measure comprises:
tuning a rate at which updates of the aggregated data are pushed to the learning machines.
9. The method as in claim 8 , further comprising:
tuning the rate as a function of feature variability.
10. The method as in claim 1 , further comprising:
extrapolating missing feature data based on the probabilistic model.
11. The method as in claim 1 , further comprising:
extrapolating future feature data based on the probabilistic model.
12. The method as in claim 1 , further comprising:
associating the predictive model with a level of confidence.
13. An apparatus, comprising:
one or more network interfaces that communicate with a computer network;
a processor coupled to the one or more network interfaces and configured to execute a process; and
a memory configured to store process executable by the processor, the process when executed operable to:
determine a plurality of machine learning features in the computer network to collect as a learning data processor;
receive data corresponding to the plurality of features;
aggregate the data;
generate a statistical analysis of all collected features;
perform a traffic reduction measure based on the statistical analysis;
learn a probabilistic model of time evolution of the received feature data; and
push the aggregated data for select features to interested learning machines associated with the computer network.
14. The apparatus as in claim 13 , wherein the process when executed to push is further operable to:
multicast each type of aggregated data within a corresponding multicast group, wherein interested learning machines join multicast groups for features in which they are interested.
15. The apparatus as in claim 13 , wherein the process when executed is further operable to:
receive, from the interested learning machines, an indication of features in which they are interested.
16. The apparatus as in claim 13 , wherein the process when executed is further operable to:
perform the traffic reduction measure by combining correlated features into a representative feature based on the statistical analysis.
17. The apparatus as in claim 13 , wherein the process when executed is further operable to:
perform the traffic reduction measure by tuning a rate at which updates of the aggregated data are pushed to the learning machines based on the statistical analysis.
18. The apparatus as in claim 13 , wherein the process when executed is further operable to:
extrapolate missing feature data based on the probabilistic model.
19. The apparatus as in claim 13 , wherein the process when executed is further operable to:
extrapolate future feature data based on the probabilistic model.
20. A tangible, non-transitory, computer-readable media having software encoded thereon, the software, when executed by a processor, operable to:
determine a plurality of machine learning features in the computer network to collect as a learning data processor;
receive data corresponding to the plurality of features;
aggregate the data;
generate a statistical analysis of all collected features;
perform a traffic reduction measure based on the statistical analysis;
learn a probabilistic model of time evolution of the received feature data; and
push the aggregated data for select features to interested learning machines associated with the computer network.