Computer-implemented method of detecting abnormal events in a telecommunication system and telecommunication system
A computer-implemented method of detecting abnormal events in a telecommunication system can include a training phase and a production phase. The training phase can include recording data events in a log file during the execution of the system and creating a standard for the data events representing a properly working environment of the telecommunication system by applying a training procedure. The production phase can include receiving at least one new data event and determining whether the at least one new data event is a normal event that matches the standard or not. If the at least one new data event matches the standard, it can be used for updating the standard. If the at least one new data event does not match the standard, it can be classified and reported as at least one an abnormal event.
1. A computer-implemented method of detecting abnormal events in a telecommunication system, the method comprising a training phase and a production phase, wherein in the training phase, the method comprises:
recording data events in a log file during the execution of the system;
creating a standard for the data events representing a properly working environment of the telecommunication system by applying a training procedure;
wherein the properly working environment is characterized by a plurality of features and a predetermined order of the features, wherein data events stored in a set of representative random samples of stored log files are used for the training procedure;
wherein the production phase comprises the steps of
receiving at least one new data event;
encoding the at least one new data event via at least one encoder;
feeding the encoded at least one new data event to a Convolutional Neural Network (CNN) for classifying the at least one new data event as a standard normal event or as an abnormal event that is an event outside of the standard; and
determining whether the at least one new data event is a normal event that matches the standard, and upon a determination that the at least one new data event matches the standard, using it for updating the standard, and upon a determination that the at least one new data event does not match the standard, classifying and reporting it as an abnormal event so that only the new data events that are categorized as an abnormal event are decoded by the decoder.
2. The method according to claim 1 , wherein a log analyzer is used for the training phase and/or in the production phase for receiving at least one log file output from the telecommunication system to be evaluated.
3. The method according to claim 1 comprising:
extracting the at least one event from the log files.
4. The method of claim 1 , comprising:
tokenizing information of the at least one new data event.
5. The method according to claim 4 , comprising:
transmitting the tokenized information of the at least one new data event to an autoencoder unit, the autoencoder unit comprising the at least one encoder and the at least one decoder.
6. The method according to claim 5 , wherein the at least one encoder is a series of stacked encoders with multiple layers or a deep encoder.
7. The method according to claim 5 , wherein the at least one decoder has one layer for decoding features from the at least one event into a format that is readable by a human.
8. The method according to claim 1 , wherein the at least one encoder encodes the at least one new data event to transform the at least one new data event from a high-dimensional space into a series of codes in a low-dimensional space.
9. The method of claim 8 , comprising:
feeding the encoded at least one new data event to a Long Short-Term Memory (LSTM).
10. The method according to claim 1 , comprising:
the CNN performing one-dimensional filtering and max-pooling of extracts of n-tuples for classifying the n-tuples as feature clusters for forming clusters of features that are classifiable under a same class by the CNN.
11. The method of claim 10 , wherein the feature clusters include multiple groups of features that includes features within the standard and features that are outside the standard.
12. A computer-implemented method of detecting abnormal events in a telecommunication system, the method comprising a training phase and a production phase, wherein in the training phase, the method comprises:
recording data events in a log file during the execution of the system;
creating a standard for the data events representing a properly working environment of the telecommunication system by applying a training procedure;
wherein the properly working environment is characterized by a plurality of features and a predetermined order of the features, wherein data events stored in a set of representative random samples of stored log files are used for the training procedure;
wherein the production phase comprises the steps of
receiving at least one new data event; and
determining whether the at least one new data event is a normal event that matches the standard, and upon a determination that the at least one new data event matches the standard, using it for updating the standard, and upon a determination that the at least one new data event does not match the standard, classifying and reporting it as an abnormal event;
tokenizing information of the at least one new data event;
transmitting the tokenized information of the at least one new data event to an autoencoder unit, the autoencoder unit comprising at least one encoder and at least one decoder, the at least one encoder comprising a series of stacked encoders with multiple layers or a deep encoder; wherein the at least one encoder encodes the at least one new data event to transform the at least one new data event from a high-dimensional space into a series of codes in a low-dimensional space;
feeding the encoded at least one new data event to a Long Short-Term Memory (LSTM);
feeding the encoded at least one new data event to a Convolutional Neural Network (CNN) for classifying the at least one new data event as a standard normal event or as an abnormal event that is an event outside of the standard, wherein only the new data events that are categorized as an abnormal event are decoded by the decoder; and
forming clusters of features that are classified by the CNN under a same class.
13. The method of claim 12 , wherein in the training phase, the at least one data event is transcribed into features that describe the telecommunication system to be evaluated.
14. A telecommunication system comprising a log analyzer, the log analyzer configured to:
detect abnormal events in the telecommunication system, the log analyzer configured to:
receive data events recorded in at least one log file during operation of the telecommunication system;
create a standard for the data events of the at least one log file representing a properly working environment of the telecommunication system by applying a training procedure;
wherein the properly working environment is characterized by a plurality of features and a predetermined order of the features, wherein data events stored in a set of representative random samples of the at least one log file are used for the training procedure;
the log analyzer also configured to
receive at least one new data event to feeding an encoded version of the at least one new data event to a Convolutional Neural Network (CNN) for classifying the at least one new data event as a standard normal event or as an abnormal event that is an event outside of the standard; and
determine whether the at least one new data event is a normal event that matches the standard, and upon a determination that the at least one new data event matches the standard, use the at least one new data event for updating the standard, and upon a determination that the at least one new data event does not match the standard, classifying and reporting the at least one new data event as at least one abnormal event so that only the new data events that are categorized as an abnormal event are decodable by at least one decoder.
15. The telecommunication system according to claim 14 , wherein the log analyzer comprises an autoencoder unit comprising a deep encoder to encode the at least one new data event for feeding the encoded version of the at least one new data event to the CNN and the at least one decoder.
16. The telecommunication system of claim 14 , comprising:
at least one encoder configured to encode the at least one new data event to transform the at least one new data event from a high-dimensional space into a series of codes in a low-dimensional space to form the encoded version of the at least one new data event for feeding to the CNN.
17. The telecommunication system of claim 16 , wherein the CNN is configured to perform one-dimensional filtering and max-pooling of extracts of n-tuples for classifying the n-tuples as feature clusters for forming clusters of features that are classifiable under a same class by the CNN.
18. The telecommunication system of claim 17 , wherein the feature clusters include multiple groups of features that includes features within the standard and features that are outside the standard.
19. The telecommunication system of claim 18 , comprising a Long Short-Term Memory (LSTM) configured to balance weights for different types of events and prevent data overfitting for the CNN.
20. The method according to claim 12 , wherein a log analyzer is used for the training phase and/or in the production phase for receiving at least one log file output from the telecommunication system to be evaluated.