RAPID PREDICTIVE ANALYSIS OF VERY LARGE DATA SETS USING AN ACTOR-DRIVEN DISTRIBUTED COMPUTATIONAL GRAPH
A system for predictive analysis of very large data sets using an actor-driven distributed computational graph, wherein a pipeline orchestrator creates and manages individual data pipelines while providing data caching to enable interactions between specific activity actors within pipelines. Each pipeline then comprises a pipeline manager that creates and manages individual activity actors and directs operations within the pipeline while reporting back to the pipeline orchestrator.
1 . A system for predictive analysis of very large data sets using an actor-driven distributed computational graph, comprising:
a pipeline orchestrator comprising a processor, a memory, and a plurality of programming instructions stored in the memory and operating on the processor, wherein the programming instructions, when operating on the processor, cause the processor to:
create a plurality of transformation pipelines each comprising at least a pipeline manager;
cache a plurality of data contexts provided by a pipeline manager;
a pipeline manager comprising a processor, a memory, and a plurality of programming instructions stored in the memory and operating on the processor, wherein the programming instructions, when operating on the processor, cause the processor to:
create a plurality of activity actors;
provide reporting data to the pipeline orchestrator;
receive at least a data context from an activity actor;
provide the data context to the pipeline orchestrator; and
an activity actor comprising a processor, a memory, and a plurality of programming instructions stored in the memory and operating on the processor, wherein the programming instructions, when operating on the processor, cause the processor to:
receive at least a set of data as a transformation input;
perform an individual transformation upon a set of data;
produce a data context based at least in part on the individual transformation;
provide the transformed set of data as a transformation output; and
provide the data context as a context output.
2 . The system of claim 1 , wherein a transformation pipeline has multiple antecedent transformation outputs used as transformation inputs into an individual transformation.
3 . The system of claim 1 , wherein a transformation output is used as a transformation input to multiple downstream transformations.
4 . The system of claim 1 , wherein the structure of a transformation pipeline is a directed graph with a plurality of individual transformations forming the nodes or vertexes of the graph and the output stream between each node forming the edges.
5 . The system of claim 2 , wherein an individual transformation within a pipeline acts as a data store and forms a queue for subsequent transformations to be performed in series.
6 . The system of claim 1 , wherein an activity actor is configured to receive a context output as an additional transformation input.
7 . The system of claim 1 , wherein a plurality of activity actors communicate directly with each other in a peer-to-peer arrangement to exchange data and messages, receiving only flow coordination messages from the pipeline manager.