Anomaly score adjustment across anomaly generators
Techniques are disclosed for generating an anomaly score for a neuro-linguistic model of input data obtained from one or more sources. According to one embodiment, generating an anomaly score comprises receiving a score indicating how often a characteristic is observed in the input data. Upon receiving the score, comparing the score with an unusual score model to determine an unusualness score and comparing the unusualness score with an anomaly score model based on one or more unusual score models to generate the anomaly score indicating an overall unusualness for the input data.
1. A method, comprising:
receiving, at a processor and at a first time, a first plurality of normalized vectors representing a first input data;
generating, via the processor, a neuro-linguistic model of the first input data based on the first plurality of normalized vectors;
generating, using the neuro-linguistic model and at a second time after the first time, a raw unusualness score for a second plurality of normalized vectors representing a second input data;
generating a normalized unusualness score based on the raw unusualness score and a plurality of historical unusualness scores;
updating the neuro-linguistic model based on the normalized unusualness score, to produce an updated neuro-linguistic model; and
sending a signal representing the updated neuro-linguistic model.
2. The method of claim 1 , wherein generating the neuro-linguistic model of the first input data includes:
clustering subsets of normalized vectors from the first plurality of normalized vectors having similar features, to form a plurality of clusters;
assigning a distinct symbol to each cluster from the plurality of clusters, to form a plurality of symbols;
identifying a plurality of recurring combinations of symbols from the plurality of symbols, each recurring combination of symbols from the plurality of recurring combinations of symbols being a word from a plurality of words; and
identifying a plurality of recurring combinations of word from the plurality of words, each recurring combination of words from the plurality of recurring combinations of words being a phrase from a plurality of phrases.
3. The method of claim 1 , wherein the second input data is data from at least one of a video data source, a supervisory control and data acquisition (SCADA) system, or a data network security system.
4. The method of claim 1 , wherein the generating the normalized unusualness score includes comparing the raw unusualness score with at least one of:
an unusual word model, an unusual syntax model, or an unusual map model.
5. The method of claim 1 , wherein generating the normalized unusualness score includes:
generating a value within a range of 0 to 1, where 0 represents a lowest unusualness and 1 represents a highest unusualness; and
associating the value with a percentile ranking representing a percentage of scores from a distribution of scores that the normalized unusualness score is one of greater than or equal to, where 0 represents a 0th percentile and 1 represents a 100th percentile.
6. The method of claim 1 , further comprising:
generating an alert based on the normalized unusualness score; and
causing transmission of a signal representing the alert.
7. The method of claim 1 , wherein the raw unusualness score represents a frequency with which a characteristic is observed in the first input data.
8. The method of claim 1 , wherein the normalized unusualness score represents an unusualness of the raw unusualness score as compared with historical raw unusualness scores.
9. A non-transitory processor-readable medium storing code representing instructions to cause a processor to:
receive, at a first time, a first plurality of normalized vectors representing a first input data;
generate a neuro-linguistic model of the first input data based on the first plurality of normalized vectors;
generate, using the neuro-linguistic model and at a second time after the first time, a raw unusualness score for a second plurality of normalized vectors representing a second input data;
generate a normalized unusualness score based on the raw unusualness score and a plurality of historical unusualness scores;
update the neuro-linguistic model based on the normalized unusualness score, to produce an updated neuro-linguistic model; and
send a signal representing the updated neuro-linguistic model.
10. The non-transitory processor-readable medium of claim 9 , wherein the instructions to cause the processor to generate the neuro-linguistic model include instructions to:
cluster subsets of normalized vectors from the first plurality of normalized vectors having similar features, to form a plurality of clusters;
assign a distinct symbol to each cluster from the plurality of clusters, to form a plurality of symbols;
identify a plurality of recurring combinations of symbols from the plurality of symbols, each recurring combination of symbols from the plurality of recurring combinations of symbols being a word from a plurality of words; and
identify a plurality of recurring combinations of word from the plurality of words, each recurring combination of words from the plurality of recurring combinations of words being a phrase from a plurality of phrases.
11. The non-transitory processor-readable medium of claim 9 , wherein the second input data is data from at least one of a video data source, a supervisory control and data acquisition (SCADA) system, or a data network security system.
12. The non-transitory processor-readable medium of claim 9 , wherein the instructions to cause the processor to generate the normalized unusualness score include instructions to compare the raw unusualness score with at least one of: an unusual word model, an unusual syntax model, or an unusual map model.
13. The non-transitory processor-readable medium of claim 9 , wherein the instructions to cause the processor to generate the normalized unusualness score include instructions to:
generate a value within a range of 0 to 1, where 0 represents a lowest unusualness and 1 represents a highest unusualness; and
associate the value with a percentile ranking representing a percentage of scores from a distribution of scores that the normalized unusualness score is one of greater than or equal to, where 0 represents a 0th percentile and 1 represents a 100th percentile.
14. The non-transitory processor-readable medium of claim 9 , further storing code representing instructions to cause the processor to:
generate an alert based on the normalized unusualness score; and
cause transmission of a signal representing the alert.
15. The non-transitory processor-readable medium of claim 9 , wherein the raw unusualness score represents a frequency with which a characteristic is observed in the first input data.
16. The non-transitory processor-readable medium of claim 9 , wherein the normalized unusualness score represents an unusualness of the raw unusualness score as compared with historical raw unusualness scores.
17. A system, comprising:
a memory; and,
a processor operatively coupled to the memory, the processor configured to:
receive, at a first time, a first plurality of normalized vectors representing a first input data;
generate a neuro-linguistic model of the first input data based on the first plurality of normalized vectors;
generate, using the neuro-linguistic model and at a second time after the first time, a raw unusualness score for a second plurality of normalized vectors representing a second input data;
generate a normalized unusualness score based on the raw unusualness score and a plurality of historical unusualness scores;
update the neuro-linguistic model based on the normalized unusualness score, to produce an updated neuro-linguistic model; and
send a signal representing the updated neuro-linguistic model.
18. The system of claim 17 , wherein the second input data is data from at least one of a video data source, a supervisory control and data acquisition (SCADA) system, or a data network security system.
19. The system of claim 17 , wherein the generating the normalized unusualness score includes comparing the raw unusualness score with at least one of:
an unusual word model, an unusual syntax model, or an unusual map model.
20. The system of claim 17 , wherein the generating the normalized unusualness score includes:
generating a value within a range of 0 to 1, where 0 represents a lowest unusualness and 1 represents a highest unusualness; and
associating the value with a percentile ranking representing a percentage of scores from a distribution of scores that the normalized unusualness score is one of greater than or equal to, where 0 represents a 0th percentile and 1 represents a 100th percentile.
21. The system of claim 17 , further comprising:
generating an alert based on the normalized unusualness score; and
causing transmission of a signal representing the alert.
22. The system of claim 17 , wherein the raw unusualness score represents a frequency with which a characteristic is observed in the first input data.
23. The system of claim 17 , wherein the normalized unusualness score represents an unusualness of the raw unusualness score as compared with historical raw unusualness scores.