Efficient transformer for content-aware anomaly detection in event sequences
A method for implementing a self-attentive encoder-decoder transformer framework for anomaly detection in event sequences is presented. The method includes feeding event content information into a content-awareness layer to generate event representations, inputting, into an encoder, event sequences of two hierarchies to capture long-term and short-term patterns and to generate feature maps, adding, in the decoder, a special sequence token at a beginning of an input sequence under detection, during a training stage, applying a one-class objective to bound the decoded special sequence token with a reconstruction loss for sequence forecasting using the generated feature maps from the encoder, and during a testing stage, labeling any event representation whose decoded special sequence token lies outside a hypersphere as an anomaly.
1. A method for implementing a self-attentive encoder-decoder transformer framework for anomaly detection in event sequences, the method comprising:
feeding event content information into a content-awareness layer to generate event representations;
inputting, into an encoder, event sequences of two hierarchies to capture long-term and short-term patterns and to generate feature maps;
adding, in the decoder, a special sequence token at a beginning of an input sequence under detection;
during a training stage, applying a one-class objective to bound the decoded special sequence token with a reconstruction loss for sequence forecasting using the generated feature maps from the encoder; and
during a testing stage, labeling any event representation whose decoded special sequence token lies outside a hypersphere as an anomaly.
2. The method of claim 1 , wherein the decoder embeds the event sequences into a latent space where anomalies are distinguishable.
3. The method of claim 1 , wherein the special sequence token represents event sequence status.
4. The method of claim 1 , wherein the encoder includes attention blocks, 1-D convolutional filters with activation functions, and MaxPool layers to downsample the inputted event sequences of the two hierarchies.
5. The method of claim 1 , wherein the decoder includes a masked self-attention layer to preserve an auto-regressive property.
6. The method of claim 1 , wherein the decoder performs a one-time interference to predict all events.
7. The method of claim 1 , wherein the input sequence under detection of the decoder includes padded zeroes inferred by a one forward procedure.
8. A non-transitory computer-readable storage medium comprising a computer-readable program for implementing a self-attentive encoder-decoder transformer framework for anomaly detection in event sequences, wherein the computer-readable program when executed on a computer causes the computer to perform the steps of:
feeding event content information into a content-awareness layer to generate event representations;
inputting, into an encoder, event sequences of two hierarchies to capture long-term and short-term patterns and to generate feature maps;
adding, in the decoder, a special sequence token at a beginning of an input sequence under detection;
during a training stage, applying a one-class objective to bound the decoded special sequence token with a reconstruction loss for sequence forecasting using the generated feature maps from the encoder; and
during a testing stage, labeling any event representation whose decoded special sequence token lies outside a hypersphere as an anomaly.
9. The non-transitory computer-readable storage medium of claim 8 , wherein the decoder embeds the event sequences into a latent space where anomalies are distinguishable.
10. The non-transitory computer-readable storage medium of claim 8 , wherein the special sequence token represents event sequence status.
11. The non-transitory computer-readable storage medium of claim 8 , wherein the encoder includes attention blocks, 1-D convolutional filters with activation functions, and MaxPool layers to downsample the inputted event sequences of the two hierarchies.
12. The non-transitory computer-readable storage medium of claim 8 , wherein the decoder includes a masked self-attention layer to preserve an auto-regressive property.
13. The non-transitory computer-readable storage medium of claim 8 , wherein the decoder performs a one-time interference to predict all events.
14. The non-transitory computer-readable storage medium of claim 8 , wherein the input sequence under detection of the decoder includes padded zeroes inferred by a one forward procedure.
15. A system for implementing a self-attentive encoder-decoder transformer framework for anomaly detection in event sequences, the system comprising:
a memory; and
one or more processors in communication with the memory configured to:
feed event content information into a content-awareness layer to generate event representations;
input, into an encoder, event sequences of two hierarchies to capture long-term and short-term patterns and to generate feature maps;
add, in the decoder, a special sequence token at a beginning of an input sequence under detection;
during a training stage, apply a one-class objective to bound the decoded special sequence token with a reconstruction loss for sequence forecasting using the generated feature maps from the encoder; and
during a testing stage, label any event representation whose decoded special sequence token lies outside a hypersphere as an anomaly.
16. The system of claim 15 , wherein the decoder embeds the event sequences into a latent space where anomalies are distinguishable.
17. The system of claim 15 , wherein the special sequence token represents event sequence status.
18. The system of claim 15 , wherein the encoder includes attention blocks, 1-D convolutional filters with activation functions, and MaxPool layers to downsample the inputted event sequences of the two hierarchies.
19. The system of claim 15 , wherein the decoder includes a masked self-attention layer to preserve an auto-regressive property.
20. The system of claim 15 , wherein the decoder performs a one-time interference to predict all events.