Multitemporal data analysis
A system for multitemporal data analysis is provided, comprising a directed computation graph service module configured to receive input data from a plurality of sources, analyze the input data to determine a best course of action for analyzing the input data, and split the input data for queueing to a general transformer service module or a decomposable service module based at least in part by analysis of the input data; a general transformer service module configured to receive data from the directed computation graph service module, and perform analysis on the received data; and a general transformer service module configured to receive data from directed computational graph module, and perform analysis on the received data.
1. A system for multitemporal data analysis, comprising:
a computing device comprising a memory, a processor, and a non-volatile data storage device;
a directed computational graph service module comprising a first plurality of programming instructions stored in the memory and operable on the processor, wherein the first plurality of programming instructions, when operating on the processor, causes the computing device to:
convert input data, comprising a combination of stored data and streaming data, into a directed computational graph, wherein:
the directed computational graph comprises nodes representing data transformations and edges representing messaging between the nodes; and
the data transformation of each node is shared among a plurality of instantiated workers, wherein the nodes, edges, and instantiated workers of the directed computational graph represent state information for processing of the input data; and
analyze the directed computational graph to determine whether to batch process or real-time process portions of the input data based on measuring shared state information among the nodes, edges, and instantiated workers of the directed computational graph;
a general transformer service module comprising a memory, a processor, and a plurality of programming instructions stored in the memory thereof and operable on the processor thereof, wherein the programmable instructions, when operating on the processor, cause the processor to:
receive batch processing data from the directed computational graph service module; and
perform batch processing of the batch processing data according to a pre-determined first data processing workflow; and
a decomposable transformer service module comprising a memory, a processor, and a plurality of programming instructions stored in the memory thereof and operable on the processor thereof, wherein the programmable instructions, when operating on the processor, cause the processor to:
receive real-time processing data from the directed computational graph service module; and
perform real-time processing of the real-time processing data according to a pre-determined second data processing workflow.
2. The system of claim 1 , wherein a function is executed based at least in part by the results of data processing by the general transformer service module and decomposable transformer service module.
3. The system of claim 1 , wherein at least a portion of the input data comes from a social media source.
4. The system of claim 1 , wherein at least a portion of the data input into the system comes from actions of a user while using an application.
5. The system of claim 1 , wherein at least a portion of the input data from a news outlet.
6. The system of claim 1 , wherein at least a portion of the input data comes from a distributed database.
7. A method for multitemporal data analysis, comprising the steps of:
converting, using a directed computation graph service module operating on a computing device, input data, comprising a combination of stored data and streaming data, into a directed computational graph, wherein:
the directed computational graph comprises nodes representing data transformations and edges representing messaging between the nodes; and
the data transformation of each node is shared among a plurality of instantiated workers, wherein the nodes, edges, and instantiated workers of the directed computational graph represent state information for processing of the input data;
analyzing, using the directed computational graph module, the directed computational graph to determine whether to batch process or real-time process portions of the input data based on measuring shared state information among the nodes, edges, and instantiated workers of the directed computational graph;
receiving batch processing data from the directed computation graph service module, using a general transformer service module operating on the computing device;
performing batch processing of the batch processing data according to a pre-determined first data processing workflow, using the general transformer service module;
receiving real-time processing data from the directed computation graph service module, using a decomposable transformer service module operating on the computing device; and
performing real-time processing of the real-time data according to a pre-determined second data processing workflow, using the decomposable transformer service module.
8. The method of claim 7 , wherein a function is executed based at least in part by the results of data processing by the general transformer service module and decomposable transformer service module.
9. The method of claim 7 , wherein at least a portion of the input data comes from a social media source.
10. The method of claim 7 , wherein at least a portion of the input data comes from actions of a user while using an application.
11. The method of claim 7 , wherein at least a portion of the input data comes from a news outlet.
12. The method of claim 7 , wherein at least a portion of the input data comes from a distributed database.