NOVELTY DETECTION SYSTEM
A graph-based novelty detection system processes a stream of observations by storing a compressed representation of the observations in a graph. Each node of the graph represents a component of an observation and a count of a number of times that component has been observed. The system calculates a novelty score based, at least in part, on a conditional probability of one or more components of the observation.
1 . A system, comprising:
at least one processor; and
at least one memory storing instructions that, in response to execution by the at least one processor, cause the system to:
store, in a graph, a first plurality of nodes indicative of an observation read from a stream of observations, wherein nodes of the first plurality of nodes correspond to pieces of information in the observation, and wherein the nodes of the first plurality of nodes comprise information indicative of a number of times one or more pieces of categorical information have been observed in the stream of observations;
calculate a score indicative of a degree to which the observation is an outlier with respect to other observations read from the stream of observations, the score calculated based at least in part on the number of times the one or more pieces of categorical information has been observed; and
send the score to a client process.
2 . The system of claim 1 , wherein the first plurality of nodes comprise an ordered sequence that corresponds to a sequence of an observation read from the stream of observations.
3 . The system of claim 2 , wherein the score is calculated based, at least in part, on a conditional probability computed based, at least in part, on a first node of the ordered sequence and a second node of the ordered sequence, the second node subsequent to the first node in the sequence.
4 . The system of claim 3 , wherein the conditional probability is computed based, at least in part, on a first number of times a first piece of categorical information has been observed in the stream of observations and on a second number of times a second piece of categorical information has been observed in the stream of observations, wherein data for the first number of times is obtained from the first node of the sequence, wherein data for the second number of times is obtained from the second node of the sequence.
5 . The system of claim 1 , the at least one memory storing further instructions that, in response to execution by the at least one processor, cause the system to:
increment a counter of a node of the first plurality of nodes, the node also included in a second plurality of nodes overlapping with the first plurality of nodes, the node representing a piece of information in the observation that corresponds to a piece of information in the second observation.
6 . The system of claim 1 , the at least one memory storing further instructions that, in response to execution by the at least one processor, cause the system to:
traverse a path through the graph, the path defined by components of the observation; and
increment counters of nodes on the path that correspond to pieces of information in the observation.
7 . The system of claim 1 , the at least one memory storing further instructions that, in response to execution by the at least one processor, cause the system to:
traverse a path defined by components of the observation.
8 . The system of claim 1 , wherein the score is calculated based, at least in part, on a probability determined based, at least in part, on the number of times the piece of categorical information has been observed and a second number of times a prior piece of categorical information has been observed in the observation.
9 . The system of claim 1 , wherein the score is adjusted based at least in part on relative information conveyed by a number of available alternative observations.
10 . The system of claim 1 , the at least one memory storing further instructions that, in response to execution by the at least one processor, cause the system to:
determine that cardinality of a node of the plurality of nodes exceeds a threshold level; and
in response to the determination, replace one or more edges of the node with a counter indicative of a number of children of the node.
11 . A computer-implemented method, comprising:
generating a graph representation based, at least in part, on a stream of observations,
wherein the graph comprises at least a first plurality of node representations,
wherein the first plurality of node representations is indicative of an observation read from the stream of observations,
wherein a node of the first plurality of node representations comprises information indicative of a number of times a piece of categorical information has been observed in the stream of observations;
calculating a score indicative of a degree to which the observation is an outlier with respect to other observations, the score based, at least in part, on the number of times the piece of categorical information has been observed; and
providing the score.
12 . The computer-implemented method of claim 11 , wherein the graph representation comprises information indicative of a tree and the tree comprises the first plurality of nodes.
13 . The computer-implemented method of claim 11 , further comprising:
incrementing a counter of a node, of the first plurality of node representations, that represents a piece of information in the observation that corresponds to a piece of information in a second observation.
14 . The computer-implemented method of claim 11 , further comprising:
traversing a path through the graph representation, the path defined by components of the observation; and
incrementing counters of node representations on the path that correspond to pieces of information in the observation.
15 . The computer-implemented method of claim 11 , wherein the score is calculated based, at least in part, on a conditional probability determined based, at least in part, on the number of times the piece of categorical information has been observed and a second number of times a prior piece of categorical information has been observed in the observation.
16 . The computer-implemented method of claim 11 , further comprising:
determining an order of the first plurality of node representations based, at least in part, on one or more of cardinality of a piece of information in the observation, a weighted graph model, or an example of an anomalous observation.
17 . A non-transitory computer-readable storage medium comprising instructions that, in response to execution by at least one processor of one or more computing devices, cause the one or more computing devices to at least:
receive data indicative of a stream of observations;
store a plurality of nodes in a graph, the plurality of nodes representative of an observation read from the stream of observations, wherein a node in the plurality of nodes comprising information indicative of a number of times a piece of information has been observed in the stream of observations;
determine a score indicative of a degree to which the observation is an outlier with respect to other observations, the score based, at least in part, on the number of times the piece of information has been observed; and
provide the score.
18 . The non-transitory computer-readable storage medium of claim 17 , comprising further instructions that, in response to execution by at least one processor of one or more computing devices, cause the one or more computing devices to at least:
traverse a path through the graph, the path defined by components of the observation; and
increment counters of nodes on the path that correspond to pieces of information in the observation.
19 . The non-transitory computer-readable storage medium of claim 17 , comprising further instructions that, in response to execution by at least one processor of one or more computing devices, cause the one or more computing devices to at least:
determine the number of times the piece of information has been observed in the stream of observations;
obtain a count of a number of siblings of a node in the first plurality of nodes;
compute a conditional probability of the piece of information occurring in the stream of observations;
compute an indicator of information content associated with the conditional probability; and
scale the indicator based, at least in part, on the count of the number of siblings.
20 . The non-transitory computer-readable storage medium of claim 19 , wherein the score is based, at least in part, on the scaled indicator.