Detection of hidden anomalies in network data
A method of identifying a hidden anomaly in a stream of time-series data. The stream of time-series data is parsed into a number of sub-streams. Each sub-stream in the number of sub-streams includes a parameter in a number of parameters. A number of policies are executed on the number of sub-streams. Executing outputs a first set of values in a current time window for each of the number of sub-streams. For the each sub-stream, a corresponding difference determination is generated by comparing, for the each sub-stream, the first set of values in the current time window to a second set of values in a past time window for the each sub-stream. When, for a sub-stream, a difference determination satisfies a threshold, an alert is returned that the hidden anomaly occurred with respect to the parameter for the selected sub-stream.
1 . A method of identifying a hidden anomaly in a stream of time-series data, the method comprising:
parsing the stream of time-series data into a plurality of sub-streams, wherein:
the stream of time-series data comprises a plurality of parameters,
each sub-stream in the plurality of sub-streams comprises a different subset of parameters in the plurality of parameters, such that each of the plurality of sub-streams is unique relative to others of the plurality of sub-streams, and
each of the plurality of parameters comprises metadata describing a corresponding unique category of data generated in an automated enterprise system;
executing a plurality of policies on the plurality of sub-streams, wherein:
a corresponding unique policy or unique policy combination in the plurality of policies is executed on each of the plurality of sub-streams, relative to other sub-streams in the plurality of sub-streams, and
executing outputs a first set of values in a current time window for each of the plurality of sub-streams;
generating, according to the corresponding unique policy or unique policy combination for the each sub-stream, a corresponding difference determination by comparing, for the each sub-stream, the first set of values in the current time window to a second set of values in a past time window for the each sub-stream,
wherein each time interval between the current time window and the past time window for the each sub-stream is different;
identifying, in a selected sub-stream in the plurality of sub-streams, that the corresponding difference determination satisfies a corresponding threshold, wherein:
the corresponding threshold applies to the selected sub-stream, and
each sub-stream in the plurality of sub-streams, including the corresponding threshold, has a different threshold relative to other sub-streams in the plurality of sub-streams; and
returning, responsive to identifying and only for the selected sub-stream, an alert that the hidden anomaly occurred with respect to a selected parameter for the selected sub-stream.
2 . The method of claim 1 , further comprising:
mitigating, automatically, the hidden anomaly.
3 . The method of claim 2 , wherein mitigating comprises:
determining a type of the hidden anomaly,
calling an escalation policy based on the type of the hidden anomaly, and
executing the escalation policy to perform mitigating the hidden anomaly.
4 . The method of claim 2 , wherein mitigating comprises one of:
restarting a program or device that generates the selected sub-stream,
stopping the program or device that generates the selected sub-stream,
performing an anti-virus scan on the program or device that generates the selected sub-stream,
executing a classification machine learning model on the selected sub-stream to classify a cause for the hidden anomaly, and
executing a diagnostic algorithm on the program or device that generates the selected sub-stream.
5 . The method of claim 1 , further comprising:
transmitting a communication signal to a user device assigned to the selected sub-stream.
6 . The method of claim 1 , wherein the plurality of sub-streams each comprise authorization permit or authorization deny determinations for accessing a plurality of different computer executable services based on the plurality of parameters.
7 . The method of claim 1 , wherein executing further comprises:
identifying a selected combination of parameters that correspond to a subset of users of a service that generates the stream of time-series data, and
adding a user group stream to the plurality of sub-streams, wherein the user group stream corresponds to the subset of users.
8 . The method of claim 7 , wherein:
the selected sub-stream comprises the user group stream, and
returning comprises returning the alert only to the subset of users.
9 . The method of claim 1 , further comprising:
passing, after identifying but prior to returning, the selected sub-stream to a feedback loop; and
determining, by the feedback loop, that the hidden anomaly corresponds to a detected event, wherein
returning is further performed responsive to the detected event.
10 . A system comprising:
a computer processor;
a data repository in communication with the computer processor and storing:
a hidden anomaly,
a stream of time-series data,
a plurality of sub-streams of the time-series data, wherein:
the stream of time-series data further comprises a plurality of parameters,
each sub-stream in the plurality of sub-streams comprises a parameter in the plurality of parameters,
each parameter in each sub-stream is a different parameter or a different combination of parameters relative to other sub-streams in the plurality of sub-streams, and
each of the plurality of parameters comprises metadata detailing a corresponding unique category of data generated in an automated enterprise system,
a plurality of policies, wherein each of the plurality of policies is different relative to other policies in the plurality of policies,
a first set of values in a current time window for each of the plurality of sub-streams,
a second set of values in a past time window for the each sub-stream,
a corresponding difference determination for each sub-stream, and
a corresponding threshold for each sub-stream,
a server controller executable by the computer processor to:
parse the stream of time-series data into the plurality of sub-streams,
execute the plurality of policies on the plurality of sub-streams, wherein:
each of the plurality of sub-streams has a corresponding unique policy or unique policy combination in the plurality of policies, relative to other sub-streams in the plurality of sub-streams, and
executing outputs the first set of values in the current time window,
generate, according to the corresponding unique policy or unique policy combination for the each sub-stream, the corresponding difference determination by comparing, for the each sub-stream, the first set of values in the current time window to the second set of values in the past time window for the each sub-stream,
wherein each time interval between the current time window and the past time window for the each sub-stream is different,
identify, in a selected sub-stream in the plurality of sub-streams, that the corresponding difference determination satisfies the corresponding threshold, wherein:
the corresponding threshold applies to the selected sub-stream, and
each sub-stream in the plurality of sub-streams, including the corresponding threshold, has a different threshold relative to other sub-streams in the plurality of sub-streams, and
return, responsive to identifying and only for the selected sub-stream, an alert that the hidden anomaly occurred with respect to a selected parameter for the selected sub-stream.
11 . The system of claim 10 , further comprising:
a mitigation service executable by the computer processor to mitigate, automatically, the hidden anomaly.
12 . The system of claim 11 , wherein mitigating comprises:
determining a type of the hidden anomaly, and
calling an escalation policy based on the type of the hidden anomaly.
13 . The system of claim 11 , wherein mitigating comprises one of:
restarting a program or device that generates the selected sub-stream,
stopping the program or device that generates the selected sub-stream,
performing an anti-virus scan on the program or device that generates the selected sub-stream,
executing a classification machine learning model on the selected sub-stream to classify a cause for the hidden anomaly, and
executing a diagnostic algorithm on the program or device that generates the selected sub-stream.
14 . The system of claim 10 , further comprising:
a communication device executable by the computer processor to transmit a communication signal to a user device assigned to the selected sub-stream.
15 . The system of claim 10 , wherein executing further comprises:
identifying a selected combination of parameters that correspond to a subset of users of a service that generates the stream of time-series data, and
adding a user group stream to the plurality of sub-streams, wherein the user group stream corresponds to the subset of users.
16 . The system of claim 15 , wherein:
the selected sub-stream comprises the user group stream, and
returning comprises returning the alert only to the subset of users.
17 . The system of claim 10 , wherein the server controller is further executable by the computer processor to:
pass, after identifying but prior to returning, the selected sub-stream to a feedback loop; and
determine, by the feedback loop, that the hidden anomaly corresponds to a detected event, wherein
returning is further performed responsive to the detected event.
18 . A method of mitigating a hidden anomaly in a stream of time-series data, the method comprising:
parsing the stream of time-series data into a plurality of sub-streams, wherein:
the stream of time-series data comprises a plurality of parameters,
each sub-stream in the plurality of sub-streams comprises a different subset of parameters in the plurality of parameters, such that each of the plurality of sub-streams is unique relative to others of the plurality of sub-streams,
each of the plurality of parameters comprises metadata detailing a corresponding unique category of data generated in an automated enterprise system, and
the stream of time-series data further comprises permit or deny decisions from authorization attempts to access a plurality of different computer-executed services,
executing a plurality of policies on the plurality of sub-streams, wherein:
a corresponding unique policy or unique policy combination in the plurality of policies is executed on each of the plurality of sub-streams, relative to other sub-streams in the plurality of sub-streams, and
executing outputs a first set of values in a current time window for each of the plurality of sub-streams;
generating, according to the corresponding unique policy or unique policy combination for the each sub-stream, a corresponding difference determination by comparing, for the each sub-stream, the first set of values in the current time window to a second set of values in a past time window for the each sub-stream,
wherein each time interval between the current time window and the past time window for the each sub-stream is different;
identifying, in a selected sub-stream in the plurality of sub-streams, that the corresponding difference determination satisfies a corresponding threshold, wherein:
the corresponding threshold applies to the selected sub-stream, and
each sub-stream in the plurality of sub-streams, including the corresponding threshold, has a different threshold relative to other sub-streams in the plurality of sub-streams; and
returning, responsive to identifying and only for the selected sub-stream, an alert that the hidden anomaly occurred with respect to a selected parameter for the selected sub-stream;
identifying a mitigation service specific to the selected sub-stream, and
mitigating the hidden anomaly by calling, by the mitigation service, an escalation policy specific to the selected sub-stream and programmed to mitigate the hidden anomaly.