Intelligent clustering systems and methods useful for domain protection
An intelligent clustering system has a dual-mode clustering engine for mass-processing and stream-processing. A tree data model is utilized to describe heterogenous data elements in an accurate and uniform way and to calculate a tree distance between each data element and a cluster representative. The clustering engine performs element clustering, through sequential or parallel stages, to cluster the data elements based at least in part on calculated tree distances and parameter values reflecting user-provided domain knowledge on a given objective. The initial clusters thus generated are fine-tuned by undergoing an iterative self-tuning process, which continues when new data is streamed from data source(s). The clustering engine incorporates stage-specific domain knowledge through stage-specific configurations. This hybrid approach combines strengths of user domain knowledge and machine learning power. Optimized clusters can be used by a prediction engine to increase prediction performance and/or by a network security specialist to identify hidden patterns.
1. An intelligent clustering system, comprising:
a processor;
a non-transitory computer-readable medium; and
stored instructions translatable by the processor to implement a mass-processing mode and a stream-processing mode;
wherein, in the mass-processing mode, the stored instructions are translatable by the processor to perform:
retrieving raw data from a data storage, the raw data comprising data elements;
preparing the data elements for clustering, the preparing including generating a tree data model for each data element and calculating a tree distance between each data element and a cluster representative utilizing the tree data model; and
providing calculated tree distances and parameter values to a clustering engine, wherein the parameter values reflect user-provided domain knowledge on a given objective, wherein the cluster engine performs an element clustering operation to cluster the data elements based at least in part on the calculated tree distances and the parameter values, and wherein the element clustering operation produces initial clusters and then fine-tunes the clusters as directed by the given objective through the parameter values;
wherein, in the stream-processing mode, the stored instructions are translatable by the processor to perform:
receiving new data streamed from a data source, the new data containing a new data element;
preparing the new data element, the preparing including generating a new tree data model for the new data element and calculating a new tree distance between the new data element and the cluster representative utilizing the new tree data model;
based at least on the new tree distance, generating a prediction for the new data element, wherein the prediction comprises an assignment of the new data element to a cluster of the clusters produced in the mass-processing mode or a new cluster containing the new data element; and
providing the prediction to the clustering engine, wherein the cluster engine performs a clustering tuning operation based at least in part on the prediction, and wherein the clustering tuning operation produces updated clusters.
2. The intelligent clustering system of claim 1 , wherein the stored instructions are further translatable by the processor to perform:
providing the updated clusters to a client device or a prediction engine, wherein the prediction engine utilizes the updated clusters to increase an accuracy rate in generating future cluster assignment predictions.
3. The intelligent clustering system of claim 1 , wherein the clustering tuning operation is an iterative self-tuning process and wherein the clustering tuning operation produces the updated clusters when a condition is met to end the iterative self-tuning process.
4. The intelligent clustering system of claim 1 , wherein the stored instructions are further translatable by the processor to perform:
providing a configuration editing module user interface to a client device, wherein the configuration editing module user interface include interactive user interface elements for describing the parameter values that reflect the user-provided domain knowledge on the given objective.
5. The intelligent clustering system of claim 1 , wherein, in the mass-processing mode, the stored instructions are further translatable by the processor to perform a sequential clustering operation in which the element clustering operation is performed sequentially through a plurality of stages, wherein each of the plurality of stages has a stage-specific configuration file containing user-provided parameter values that reflect stage-specific objective, wherein, in each stage, a first clustering result undergoes a user view and a second clustering result is generated based on information received from the user review, wherein a significance rule is applied to the second clustering result, wherein any cluster that does not meet the significance rule is carried over to a next stage, and wherein the sequential clustering operation ends when all the data elements have been processed or when the plurality of stages is complete.
6. The intelligent clustering system of claim 1 , wherein, in the mass-processing mode, the stored instructions are further translatable by the processor to perform a parallel clustering operation in which the element clustering operation is performed parallelly through multiple stages, wherein each of the multiple stages has a stage-specific configuration file containing user-provided parameter values that reflect stage-specific objective, wherein, in each stage, a first clustering result undergoes a user view and a second clustering result is generated based on information received from the user review, and wherein a significance rule is applied to the second clustering result such that the parallel clustering operation outputs only clusters that meet the significance rule in view of the stage-specific objective.
7. The intelligent clustering system of claim 1 , wherein the new data element comprises a plurality of features, wherein the new tree data model comprises a root node representing a feature vector having an ordered collection of components representing the plurality of features, wherein the root node points, through relationship paths, to a set of user-defined features, each of which points, through additional relationship paths, to a set of sub-trees of user-defined features, wherein each node in the new tree data model except the root node is associated with a weight and a distance function, and wherein the plurality of features, the relationship paths, the weight, and the distance function are defined in the parameter values that reflect the user-provided domain knowledge on the given objective.
8. A method, comprising:
performing, by an intelligent clustering system in a mass-processing mode, the intelligent clustering system having a processor and a non-transitory computer-readable medium:
retrieving raw data from a data storage, the raw data comprising data elements;
preparing the data elements for clustering, the preparing including generating a tree data model for each data element and calculating a tree distance between each data element and a cluster representative utilizing the tree data model; and
providing calculated tree distances and parameter values to a clustering engine, wherein the parameter values reflect user-provided domain knowledge on a given objective, wherein the cluster engine performs an element clustering operation to cluster the data elements based at least in part on the calculated tree distances and the parameter values, and wherein the element clustering operation produces initial clusters and then fine-tunes the clusters as directed by the given objective through the parameter values; and
performing, by the intelligent clustering system in a stream-processing mode:
receiving new data streamed from a data source, the new data containing a new data element;
preparing the new data element, the preparing including generating a new tree data model for the new data element and calculating a new tree distance between the new data element and the cluster representative utilizing the new tree data model;
based at least on the new tree distance, generating a prediction for the new data element, wherein the prediction comprises an assignment of the new data element to a cluster of the clusters produced in the mass-processing mode or a new cluster containing the new data element; and
providing the prediction to the clustering engine, wherein the cluster engine performs a clustering tuning operation based at least in part on the prediction, and wherein the clustering tuning operation produces updated clusters.
9. The method according to claim 8 , further comprising:
providing the updated clusters to a client device or a prediction engine, wherein the prediction engine utilizes the updated clusters to increase an accuracy rate in generating future cluster assignment predictions.
10. The method according to claim 8 , wherein the clustering tuning operation is an iterative self-tuning process and wherein the clustering tuning operation produces the updated clusters when a condition is met to end the iterative self-tuning process.
11. The method according to claim 8 , further comprising:
providing a configuration editing module user interface to a client device, wherein the configuration editing module user interface include interactive user interface elements for describing the parameter values that reflect the user-provided domain knowledge on the given objective.
12. The method according to claim 8 , further comprising:
in the mass-processing mode, performing a sequential clustering operation in which the element clustering operation is performed sequentially through a plurality of stages, wherein each of the plurality of stages has a stage-specific configuration file containing user-provided parameter values that reflect stage-specific objective, wherein, in each stage, a first clustering result undergoes a user view and a second clustering result is generated based on information received from the user review, wherein a significance rule is applied to the second clustering result, wherein any cluster that does not meet the significance rule is carried over to a next stage, and wherein the sequential clustering operation ends when all the data elements have been processed or when the plurality of stages is complete.
13. The method according to claim 8 , further comprising:
performing, in the mass-processing mode, a parallel clustering operation in which the element clustering operation is performed parallelly through multiple stages, wherein each of the multiple stages has a stage-specific configuration file containing user-provided parameter values that reflect stage-specific objective, wherein, in each stage, a first clustering result undergoes a user view and a second clustering result is generated based on information received from the user review, and wherein a significance rule is applied to the second clustering result such that the parallel clustering operation outputs only clusters that meet the significance rule in view of the stage-specific objective.
14. The method according to claim 8 , wherein the new data element comprises a plurality of features, wherein the new tree data model comprises a root node representing a feature vector having an ordered collection of components representing the plurality of features, wherein the root node points, through relationship paths, to a set of user-defined features, each of which points, through additional relationship paths, to a set of sub-trees of user-defined features, wherein each node in the new tree data model except the root node is associated with a weight and a distance function, and wherein the plurality of features, the relationship paths, the weight, and the distance function are defined in the parameter values that reflect the user-provided domain knowledge on the given objective.
15. A computer program product comprising a non-transitory computer-readable medium storing instructions translatable by a processor to perform:
in a mass-processing mode:
retrieving raw data from a data storage, the raw data comprising data elements;
preparing the data elements for clustering, the preparing including generating a tree data model for each data element and calculating a tree distance between each data element and a cluster representative utilizing the tree data model; and
providing calculated tree distances and parameter values to a clustering engine, wherein the parameter values reflect user-provided domain knowledge on a given objective, wherein the cluster engine performs an element clustering operation to cluster the data elements based at least in part on the calculated tree distances and the parameter values, and wherein the element clustering operation produces initial clusters and then fine-tunes the clusters as directed by the given objective through the parameter values;
in a stream-processing mode:
receiving new data streamed from a data source, the new data containing a new data element;
preparing the new data element, the preparing including generating a new tree data model for the new data element and calculating a new tree distance between the new data element and the cluster representative utilizing the new tree data model;
based at least on the new tree distance, generating a prediction for the new data element, wherein the prediction comprises an assignment of the new data element to a cluster of the clusters produced in the mass-processing mode or a new cluster containing the new data element; and
providing the prediction to the clustering engine, wherein the cluster engine performs a clustering tuning operation based at least in part on the prediction, and wherein the clustering tuning operation produces updated clusters; and
providing the updated clusters to a client device or a prediction engine, wherein the prediction engine utilizes the updated clusters to increase an accuracy rate in generating future cluster assignment predictions.
16. The computer program product of claim 15 , wherein the clustering tuning operation is an iterative self-tuning process and wherein the clustering tuning operation produces the updated clusters when a condition is met to end the iterative self-tuning process.
17. The computer program product of claim 15 , wherein the instructions are further translatable by the processor to perform:
providing a configuration editing module user interface to a client device, wherein the configuration editing module user interface include interactive user interface elements for describing the parameter values that reflect the user-provided domain knowledge on the given objective.
18. The computer program product of claim 15 , wherein the instructions are further translatable by the processor to perform, in the mass-processing mode, a sequential clustering operation in which the element clustering operation is performed sequentially through a plurality of stages, wherein each of the plurality of stages has a stage-specific configuration file containing user-provided parameter values that reflect stage-specific objective, wherein, in each stage, a first clustering result undergoes a user view and a second clustering result is generated based on information received from the user review, wherein a significance rule is applied to the second clustering result, wherein any cluster that does not meet the significance rule is carried over to a next stage, and wherein the sequential clustering operation ends when all the data elements have been processed or when the plurality of stages is complete.
19. The computer program product of claim 15 , wherein the instructions are further translatable by the processor to perform, in the mass-processing mode, a parallel clustering operation in which the element clustering operation is performed parallelly through multiple stages, wherein each of the multiple stages has a stage-specific configuration file containing user-provided parameter values that reflect stage-specific objective, wherein, in each stage, a first clustering result undergoes a user view and a second clustering result is generated based on information received from the user review, and wherein a significance rule is applied to the second clustering result such that the parallel clustering operation outputs only clusters that meet the significance rule in view of the stage-specific objective.
20. The computer program product of claim 15 , wherein the new data element comprises a plurality of features, wherein the new tree data model comprises a root node representing a feature vector having an ordered collection of components representing the plurality of features, wherein the root node points, through relationship paths, to a set of user-defined features, each of which points, through additional relationship paths, to a set of sub-trees of user-defined features, wherein each node in the new tree data model except the root node is associated with a weight and a distance function, and wherein the plurality of features, the relationship paths, the weight, and the distance function are defined in the parameter values that reflect the user-provided domain knowledge on the given objective.