Rapid predictive analysis of very large data sets using the distributed computational graph
A system for fully integrated collection of business impacting data, analysis of that data and generation of both analysis driven business decisions and analysis driven simulations of alternate candidate business actions has been devised and reduced to practice. This business operating system may be used to monitor and predictively warn of events that impact the security of business infrastructure and may also be employed to monitor client-facing services supported by both software and hardware to alert in case of reduction or failure and also predict deficiency, service reduction or failure based on current event data.
1. A system comprising:
a distributed computing cluster comprising a first plurality of computer systems, a second plurality of computer systems, and a third plurality of computer systems,
wherein the first plurality of computer systems is configured to execute software instructions that:
receive a first data stream,
process at least a portion of the first data stream using a first transformation pipeline,
wherein the first transformation pipeline generates a database query,
and cause the database query to be transmitted to the second plurality of computer systems,
wherein the second plurality of computer systems is configured to execute software instructions that:
receive a second data stream and store, in a time-series database, at least a portion of the data received in the second data stream,
receive the database query, and
cause at least a portion of any data responsive to the database query in the time-series database to be transmitted to the third plurality of computer systems as a result of the database query,
wherein the third plurality of computer systems is configured to execute software instructions that:
receive a third data stream,
receive the result of the database query,
process at least a portion of the third data stream using a second transformation pipeline to generate first pipeline output messages,
wherein the third plurality of computer systems provides at least a portion of the result of the database query as an input to the second transformation pipeline, and transmit the first pipeline output messages.
2. The system of claim 1 , wherein the first plurality of computer systems and the third plurality of computer systems are identical.
3. The system of claim 1 , wherein at least one of the first data stream, the second data stream, and the third data stream is received from an input feed.
4. The system of claim 1 , wherein the first plurality of computer systems and the second plurality of computer systems are not identical pluralities.
5. The system of claim 1 , wherein the first plurality of computer systems and the second plurality of computer systems are distinct pluralities.
6. The system of claim 1 , wherein the second transformation pipeline is configured to output a determination about whether one or more authentication credentials has been presented by an unauthorized actor.
7. The system of claim 1 ,
wherein the distributed computing cluster further comprises a fourth plurality of computer systems, and
wherein the fourth plurality of computer systems executes software instructions that process the first pipeline output messages to output a determination about whether one or more authentication credentials has been presented by an unauthorized user.
8. The system of claim 1 , wherein the second transformation pipeline is configured to output a determination about whether one or more authentication credentials has been forged.
9. The system of claim 1 ,
wherein the distributed computing cluster further comprises a fourth plurality of computer systems, and
wherein the fourth plurality of computer systems executes software instructions that process the first pipeline output messages to output a determination about whether one or more authentication credentials has been forged.
10. The system of claim 1 , wherein the second transformation pipeline is configured to output a determination about whether one or more authentication credentials has been accessed by an unauthorized actor.
11. The system of claim 1 ,
wherein the distributed computing cluster further comprises a fourth plurality of computer systems, and
wherein the fourth plurality of computer systems executes software instructions that process the first pipeline output messages to output a determination about whether one or more authentication credentials has accessed by an unauthorized actor.
12. The system of claim 1 , wherein the second transformation pipeline is configured to output a prediction about an impending security exploit.
13. The system of claim 12 , wherein the result of the database query includes information about one or more sign-on attempts.
14. The system of claim 13 , wherein the second plurality of computer systems utilizes information about network address ranges associated with an organization known to infiltrate authentication controllers to locate the result of the database query.
15. The system of claim 1 , wherein the second transformation pipeline is configured to output a prediction about future events.
16. The system of claim 1 ,
wherein the distributed computing cluster further comprises a fourth plurality of computer systems, and
wherein the fourth plurality of computer systems executes software instructions that process the first pipeline output messages to output a prediction about future events.