DISCOVERY OF UNUSUAL, UNEXPECTED, OR ANOMALOUS INFORMATION AND TRENDS IN HIGH THROUGHPUT DATA STREAMS AND DATABASES USING PROBABILITSTIC SURPRISAL CONTEXT FILTERS
A method, system, and computer program product for detecting anomalous events from a data input comprising a plurality of events. The method comprising the steps of: selecting at least one filter selecting for context data determined to be probabilistically present within a specified degree of certainty in the data input; comparing the data input to the selected at least one filter; discarding the events from the data input that are the same as the context data for which the at least one filter selects; and storing in a repository the events remaining in the data input as anomalous events.
1 . A method for detecting anomalous events from a data input comprising a plurality of events, comprising the steps of:
a computer selecting at least one filter selecting for context data determined to be probabilistically present within a specified degree of certainty in the data input;
the computer comparing the data input to the selected at least one filter;
the computer discarding the events from the data input that are the same as the context data for which the at least one filter selects; and
the computer storing in a repository the events remaining in the data input as anomalous events.
2 . The method of claim 1 , wherein if more than one filter is present, each of the filters have context data determined to be probabilistically present at different specified degrees of certainty in the data input.
3 . The method of claim 1 , wherein the data input is a data stream.
4 . The method of claim 1 , wherein the data input is data stored in a database.
5 . The method of claim 1 , further comprising the step of the computer sending a notification to a user regarding the anomalous events.
6 . A computer program product for detecting anomalous events from a data input comprising a plurality of events, the computer program product comprising:
one or more computer-readable, tangible storage devices;
program instructions, stored on at least one of the one or more storage devices, to select at least one filter selecting for context data determined to be probabilistically present within a specified degree of certainty in the data input; program instructions, stored on at least one of the one or more storage devices, to compare the data input to the selected at least one filter;
program instructions, stored on at least one of the one or more storage devices, to discard the events from the data input that are the same as the context data for which the at least one filter selects; and
program instructions, stored on at least one of the one or more storage devices, to store in a repository the events remaining in the data input as anomalous events.
7 . The computer program product of claim 6 , wherein if more than one filter is present, each of the filters have context data determined to be probabilistically present at different specified degrees of certainty in the data input.
8 . The computer program product of claim 6 , wherein the data input is a data stream.
9 . The computer program product of claim 6 , wherein the data input is data stored in a database.
10 . The computer program product of claim 6 , further comprising program instructions, stored on at least one of the one or more storage devices, to send a notification to a user regarding the anomalous events.
11 . A system for detecting anomalous events from a data input comprising a plurality of events, the system comprising:
one or more processors, one or more computer-readable memories and one or more computer-readable, tangible storage devices;
program instructions, stored on at least one of the one or more storage devices for execution by at least one of the one or more processors via at least one of the one or more memories, to select at least one filter selecting for context data determined to be probabilistically present within a specified degree of certainty in the data input; program instructions, stored on at least one of the one or more storage devices for execution by at least one of the one or more processors via at least one of the one or more memories, to compare the data input to the selected at least one filter;
program instructions, stored on at least one of the one or more storage devices for execution by at least one of the one or more processors via at least one of the one or more memories, to discard the events from the data input that are the same as the context data for which the at least one filter selects; and
program instructions, stored on at least one of the one or more storage devices for execution by at least one of the one or more processors via at least one of the one or more memories, to store in a repository the events remaining in the data input as anomalous events.
12 . The system of claim 11 , wherein if more than one filter is present, each of the filters have context data determined to be probabilistically present at different specified degrees of certainty in the data input.
13 . The system of claim 11 , wherein the data input is a data stream.
14 . The system of claim 11 , wherein the data input is data stored in a database.
15 . The system of claim 11 , further comprising program instructions, stored on at least one of the one or more storage devices for execution by at least one of the one or more processors via at least one of the one or more memories, to send a notification to a user regarding the anomalous events.