Method and apparatus for analysis and classification of high dimensional data sets
A method executable via operation of configured processing circuitry may include constructing a mutual information graph for categorical data with respect to observed attributes of a plurality of entities described in terms of respective ones of the observed attributes by the categorical data, determining a clique tree correlating attributes having at least a threshold level of mutual dependence among the observed attributes, and determining a normality rating for an entity relative to the plurality of entities based on the clique tree.
1. An apparatus comprising processing circuitry configured to execute instructions that, when executed, cause the apparatus to:
construct a mutual information graph for categorical data with respect to observed attributes of a plurality of entities described in terms of respective ones of the observed attributes by the categorical data, wherein each observed attribute corresponds to an attribute node of the mutual information graph, wherein the attribute nodes of the mutual information graph are associated via links between the attribute nodes, and wherein the links between the attribute nodes are weighted based on a degree of dependency between the attribute nodes;
determine a clique tree correlating the observed attributes, wherein being configured to determine the clique tree comprises being configured to:
prune selected links of the mutual information graph based on weightings of the selected links and application of a link weight retention threshold to form a pruned mutual information graph;
identify a chordless cycle within the pruned mutual information graph, the chordless cycle being defined as having no attribute node on a periphery of the pruned mutual information graph with a direct link to a non-adjacent attribute node, and
in response to identifying the chordless cycle within the pruned mutual information graph, introduce at least one non-adjacent link to an attribute node on the periphery of the pruned mutual information graph, the non-adjacent link being a link between the attribute node on the periphery of the pruned mutual information graph and a different attribute node that is not directly linked to the attribute node on the periphery of the pruned mutual information graph; and
determine a normality rating for an entity relative to the plurality of entities based on the clique tree.
2. The apparatus of claim 1 , wherein the categorical data is high dimensional data, and wherein determining the normality rating comprises determining a joint probability distribution to determine a probability of the entity based on lower dimension subsets of the observed attributes.
3. The apparatus of claim 2 , wherein the entity is described by a combination of attributes, and the combination of attributes has not been previously observed.
4. The apparatus of claim 2 , wherein determining the joint probability distribution comprises determining a ratio of probabilities of cliques defined by common attribute groupings to probabilities of overlapping elements from the cliques.
5. The apparatus of claim 1 , wherein the plurality of entities are each associated with a particular classification of entities, and wherein the normality rating defines a degree to which the entity fits within the particular classification.
6. The apparatus of claim 1 , wherein determining the clique tree comprises enabling a user to adjust the pruned mutual information graph.
7. The apparatus of claim 1 , wherein the processing circuitry is further configured to automatically determine the link weight retention threshold by partitioning the categorical data randomly into a training data set and a test data set, and optimizing the link weight retention threshold by maximizing a product of probabilities of the training data set and the test data set.
8. The apparatus of claim 1 , wherein the processing circuitry is further configured to generate an output to a user terminal, the output indicating whether the entity is an anomaly, a classification of the entity, or an index of data to which the entity is similar.
9. The apparatus of claim 8 , wherein the output comprises an alarm, an alert, or an instruction to take an action relative to the entity.
10. A method executable via operation of configured processing circuitry, the method comprising:
constructing a mutual information graph for categorical data with respect to observed attributes of a plurality of entities described in terms of respective ones of the observed attributes by the categorical data, wherein each observed attribute corresponds to an attribute node of the mutual information graph, wherein the attribute nodes of the mutual information graph are associated via links between the attribute nodes, and wherein the links between the attribute nodes are weighted based on a degree of dependency between the attribute nodes;
determining a clique tree correlating the observed attributes having at least a threshold level of mutual dependence, wherein determining the clique tree includes:
prune selected links of the mutual information graph based on weightings of the selected links and application of a link weight retention threshold to form a pruned mutual information graph,
identifying a chordless cycle within the pruned mutual information graph, the chordless cycle being defined as having no attribute node on a periphery of the pruned mutual information graph with a direct link to a non-adjacent attribute node, and
in response to identifying the chordless cycle within the pruned mutual information graph, introducing at least one non-adjacent link to an attribute node on the periphery of the pruned mutual information graph, the non-adjacent link being a link between the attribute node on the periphery of the pruned mutual information graph and a different attribute node that is not directly linked to the attribute node on the peripherv of the pruned mutual information graph; and
determining a normality rating for an entity relative to the plurality of entities based on the clique tree.
11. The method of claim 10 , wherein the categorical data is high dimensional data, and wherein determining the normality rating comprises determining a joint probability distribution to determine a probability of the entity based on lower dimension subsets of the observed attributes.
12. The method of claim 11 , wherein the entity is described by a combination of attributes, and the combination of attributes has not been previously observed.
13. The method of claim 11 , wherein determining the joint probability distribution comprises determining a ratio of probabilities of cliques defined by common attribute groupings to probabilities of overlapping elements from the cliques.
14. The method of claim 10 , wherein the plurality of entities are each associated with a particular classification of entities, and wherein the normality rating defines a degree to which the entity fits within the particular classification.
15. The method of claim 10 , wherein determining the clique tree comprises enabling a user to adjust the pruned mutual information graph.
16. The method of claim 10 , further comprising automatically determining the link weight retention threshold by partitioning the categorical data randomly into a training data set and a test data set, and optimizing the link weight retention threshold by maximizing a product of probabilities of the training data set and the test data set.
17. The method of claim 10 , further comprising generating an output to a user terminal, the output indicating whether the entity is an anomaly, a classification of the entity, or an index of data to which the entity is similar.
18. The method of claim 17 , wherein the output comprises an alarm, an alert, or an instruction to take an action relative to the entity.
19. A method executable via operation of configured processing circuitry, the method comprising:
utilizing a correlation metric to construct an input graph with respect to observed attributes of a plurality of entities described in terms of respective ones of the observed attributes in categorical data, wherein each observed attribute corresponds to an attribute node of the input graph, wherein the attribute nodes of the input graph are associated via links between the attribute nodes, and wherein the links between the attribute nodes are weighted based on a degree of dependency between the attribute nodes;
determining a clique tree defining entity groupings with correlated observed attributes based on the correlation metric, wherein determining the clique tree includes:
prune selected links of the input graph based on weightings of the selected links and application of a link weight retention threshold to form a pruned input graph,
identifying a chordless cycle within the pruned input graph, the chordless cycle being defined as having no attribute node on a periphery of the pruned input graph with a direct link to a non-adjacent attribute node, and
in response to identifying the chordless cycle within the pruned input graph, introducing at least one non-adjacent link to an attribute node on the periphery of the pruned input graph, the non-adjacent link being a link between the attribute node on the periphery of the pruned input graph and a different attribute node that is not directly linked to the attribute node on the periphery of the pruned input graph; and
determining a normality rating for an entity relative to the plurality of entities based on the clique tree.
20. The method of claim 19 , wherein utilizing the correlation metric comprises constructing the input graph as a mutual information graph with respect to observed attributes of the plurality of entities.