MULTITEMPORAL DATA ANALYSIS
A system for multitemporal data analysis is provided, comprising a directed computation graph service module configured to receive input data from a plurality of sources, analyze the input data to determine a best course of action for analyzing the input data, and split the input data for queueing to a general transformer service module or a decomposable service module based at least in part by analysis of the input data; a general transformer service module configured to receive data from the directed computation graph service module, and perform analysis on the received data; and a general transformer service module configured to receive data from directed computational graph module, and perform analysis on the received data.
1 . A system for multitemporal data analysis, comprising:
a computing device comprising a memory, a processor, and a non-volatile data storage device;
a graph stack service module comprising a first plurality of programming instructions stored in the memory and operable on the processor, wherein the first plurality of programming instructions, when operating on the processor, causes the computing device to:
convert an input data stream into a directed graph;
send the directed graph to a distributed compute graph comprising a data pipeline service and configured to perform computations based on the structure and content of the directed graph;
wherein the pipeline service moves graph-based data among a plurality of instantiated workers, wherein the nodes, edges, and instantiated workers of the directed compute graph represent state information for processing of the input data;
a general transformer service module comprising a memory, a processor, and a plurality of programming instructions stored in the memory thereof and operable on the processor thereof, wherein the programmable instructions, when operating on the processor, cause the processor to:
receive batch processing data from the directed computational graph service module; and
perform batch processing of the batch processing data according to a pre-determined first data processing workflow; and
a decomposable transformer service module comprising a memory, a processor, and a plurality of programming instructions stored in the memory thereof and operable on the processor thereof, wherein the programmable instructions, when operating on the processor, cause the processor to:
receive real-time processing data from the directed computational graph service module; and
perform real-time processing of the real-time processing data according to a pre-determined second data processing workflow.
2 . The system of claim 1 , wherein a function is executed based at least in part by the results of data processing by the general transformer service module and decomposable transformer service module.
3 . The system of claim 1 , wherein at least a portion of the input data comes from a social media source.
4 . The system of claim 1 , wherein at least a portion of the data input into the system comes from actions of a user while using an application.
5 . The system of claim 1 , wherein at least a portion of the input data from a news outlet.
6 . The system of claim 1 , wherein at least a portion of the input data comes from a distributed database.
7 . A method for multitemporal data analysis, comprising the steps of:
converting an input data stream into a directed graph;
sending the directed graph to a distributed compute graph comprising a data pipeline service and configured to perform computations based on the structure and content of the directed graph;
moving, using the pipeline service, graph-based data among a plurality of instantiated workers, wherein the nodes, edges, and instantiated workers of the directed compute graph represent state information for processing of the input data;
receiving batch processing data from the directed computation graph service module, using a general transformer service module operating on the computing device;
performing batch processing of the batch processing data according to a pre-determined first data processing workflow, using the general transformer service module;
receiving real-time processing data from the directed computation graph service module, using a decomposable transformer service module operating on the computing device; and
performing real-time processing of the real-time data according to a pre-determined second data processing workflow, using the decomposable transformer service module.
8 . The method of claim 7 , wherein a function is executed based at least in part by the results of data processing by the general transformer service module and decomposable transformer service module.
9 . The method of claim 7 , wherein at least a portion of the input data comes from a social media source.
10 . The method of claim 7 , wherein at least a portion of the input data comes from actions of a user while using an application.
11 . The method of claim 7 , wherein at least a portion of the input data comes from a news outlet.
12 . The method of claim 7 , wherein at least a portion of the input data comes from a distributed database.