Advanced intelligence engine
An advanced intelligence engine (AIE) for use in identifying what may be complex events or developments on one or more data platforms or networks from various types of structured or normalized data generated by one or more disparate data sources. The AIE may conduct one or more types of quantitative, correlative, behavioral and corroborative analyses to detect events from what may otherwise be considered unimportant or non-relevant information spanning one or more time periods. Events generated by the AIE may be passed to an event manager to determine whether further action is required such as reporting, remediation, and the like.
1. A method for use in monitoring one or more platforms of one or more data systems, comprising:
receiving, at a processor, structured data generated by one or more platforms over at least one communications network; and
analyzing, at the processor using a first rule block, at least some of the data, wherein the analyzing includes:
identifying, at the processor, a first portion of the structured data;
obtaining reference data that is at least partially derived from the structured data, wherein the obtaining reference data comprises:
accessing, by the processor, a linking relationship object in the first rule block to identify a data field in the structured data;
extracting, by the processor, a content of the data field from the structured data:
identifying facts based upon their relation to the content; and
generating reference data from the facts;
evaluating, using at least a first expression, the first portion of the structured data in view of the reference data to determine whether a result is one of at least first and second outcomes; and
evaluating the result to determine an event of interest.
2. The method of claim 1 , wherein the reference data is at least partially derived from the first portion of the structured data.
3. The method of claim 1 , wherein the step of evaluating using at least the first expression includes:
generating one or more statistics from the first portion of the structured data; and
assessing the one or more generated statistics in view of the reference data to determine whether a result is one of at least the first and second outcomes.
4. The method of claim 3 , wherein the reference data comprises one or more statistics that are at least partially generated from the first portion of the structured data.
5. The method claim 3 , wherein the step of assessing includes determining whether the one or more generated statistics have a specified relation relative to the reference data.
6. The method of claim 3 , wherein the one or more generated statistics include at least one of an average of at least some of the first portion of the structured data and an aggregate of at least some of the first portion of the structured data.
7. The method of claim 3 , wherein the one or more generated statistics are generated from structured data corresponding to a first period of time, and wherein the reference data is generated from one or more data corresponding to a second period of time.
8. The method of claim 7 , wherein the first and second periods of time are the same.
9. The method of claim 7 , wherein the first and second periods of time are at least partially different.
10. The method of claim 1 , wherein the reference data includes a whitelist of approved processes or programs, and wherein the step of evaluating using at least the first expression includes:
determining whether processes or programs identified in the first portion of the structured data are found in the whitelist of approved processes or programs.
11. The method of claim 10 , further comprising:
profiling the first portion of structured data to generate the whitelist.
12. The method of claim 1 , further including:
identifying, at the processor, a second portion of the structured data that is at least partially different from the first portion of the structured data, wherein the reference data comprises statistics of the second portion of the structured data.
13. The method of claim 12 , wherein the first portion of the structured data corresponds to a first period of time, and wherein the second portion of the structured data corresponds to a second period of time.
14. The method of claim 13 , wherein the first and second periods of time are the same.
15. The method of claim 13 , wherein the first and second periods of time are at least partially different.
16. The method of claim 1 , further comprising during the step of evaluating using at least the first expression:
determining that the result is the first outcome; and
generating, in response to the step of determining, an object, wherein the object is analyzed to determine an event of interest.
17. The method of claim 1 , wherein the analyzing further includes:
evaluating, using at least a second expression, the first portion of the structured data in view of reference data to determine whether a result is one of at least first and second outcomes, wherein the step of evaluating the result includes evaluating the first expression evaluation result in view of the second expression evaluation result.
18. The method of claim 17 , wherein the reference data in the step of evaluating using at least the first expression is the same as the reference data in the step of evaluating using at least the second expression.
19. The method of claim 17 , wherein the reference data in the step of evaluating using at least the first expression is different than the reference data in the step of evaluating using at least the second expression.
20. The method of claim 17 , wherein the step of evaluating the result includes:
determining that the result of one of the step of evaluating using at least the first expression or the step of evaluating using at least the second expression is the first outcome.
21. The method of claim 20 , further comprising in response to the step of determining:
generating an object, wherein the object is analyzed to determine an event of interest.
22. The method of claim 17 , wherein the step of evaluating the result includes:
determining that the result of both of the steps of evaluating using at least the first expression and the step of evaluating using at least the second expression is the first outcome.
23. The method of claim 22 , further comprising in response to the step of determining:
generating an object, wherein the object is analyzed to determine an event of interest.
24. The method of claim 1 , further comprising:
determining, from the step of evaluating the result, whether a result is one of at least first and second outcomes;
depending upon the determining, evaluating, at the processor using a second rule block, at least some of the structured data; and
determining, from the evaluating of the second rule block, whether a result is one of at least first and second outcomes, wherein the results are analyzed to determine an event of interest.
25. A system for use in monitoring one or more platforms of one or more data systems, comprising:
a processor; and
a non-transitory computer readable medium logically connected to the processor and comprising a set of computer readable instructions that are executable by the processor to:
receive structured data generated by one or more platforms over at least one communications network;
obtain reference data that is at least partially derived from the structured data, wherein the set of computer readable instructions are executable by the processor to obtain the reference data by:
accessing a linking relationship object in the first rule block to identify a data field in the structured data;
extracting a content of the data field from the structured data;
identifying facts based upon their relation to the content; and
generating reference data from the facts; and
analyze, using a first rule block, at least some of the data, by:
identifying, at the processor, a first portion of the structured data;
evaluating, using at least a first expression, the first portion of the structured data in view of the reference data to determine that a result is a first of at least first and second outcomes; and
generating, in response to the step of determining, an object, wherein the object is analyzed to determine an event of interest.
26. A method for use in monitoring one or more platforms of one or more data systems, comprising:
receiving, at a processor, structured data generated by one or more platforms over at least one communications network; and
analyzing, at the processor using a first rule block, at least some of the data, wherein the analyzing includes:
identifying a first portion of the structured data;
obtaining reference data that is at least partially derived from the structured data, wherein the obtaining reference data comprises:
accessing, by the processor, a linking relationship object in the first rule block to identify a data field in the structured data;
extracting, by the processor, a content of the data field from the structured data:
identifying facts based upon their relation to the content; and
generating reference data from the facts;
first evaluating, using at least a first expression, the first portion of the structured data in view of the reference data to determine whether a result is one of at least first and second outcomes, wherein the evaluating includes:
ascertaining a specified relation from the first expression; and
determining whether the first portion of the structured data has the specified relation relative to the reference data; and
analyzing the result to determine an event of interest.