Highly scalable distributed connection interface for data capture from multiple network service sources
A highly scalable distributed connection interface for data capture from multiple network service sources, comprising a connector module wherein, the connector module retrieves a plurality of operational data from a plurality of network data sources; employs a plurality of application programming interface routines to communicate with the plurality of operational data sources; accepts a plurality of analysis parameters and control commands directly from human interface devices or from one or more command and control storage devices; and specifies the action or actions to be taken on the retrieved operational data.
1. A system for highly scalable distributed connection interface for data capture from multiple network service sources comprising:
a computing device comprising a memory, a processor, and a plurality of programming instructions:
a connector module comprising connections to a plurality of network data sources and a portion of the plurality of programming instructions configured to cause the computing device to:
retrieve a plurality of operational data from each of the plurality of network data sources;
accept a plurality of analysis parameters and control commands from a user of the system, the plurality of analysis parameters and control commands comprising:
instructions to either passively monitor data or actively retrieve the plurality of operational data from each of the plurality of network data sources, depending on the each of the plurality network data sources; and
send the plurality of operational data and the plurality of analysis parameters and control commands to a directed computational graph module; and
the directed computational graph module comprising a directed computational graph comprising nodes representing data transformations, edges representing message outputs, and a portion of the plurality of programming instructions configured to cause the computing device to:
receive the plurality of operational data and the plurality of analysis parameters and control commands from the connector module;
transform, using a decomposable transformer service module, the plurality of operational data into a format;
receive the transformed plurality of operational data from the decomposable transformer service module; and
update the nodes of the directed computational graph to represent the data transformations, and update the edges of the directed computational graph to represent transformed message outputs; and
the decomposable transformer service module comprising a portion of the plurality of programming instructions configured to cause the computing device to:
receive the plurality of operational data and direction from the directed computational graph module; and
convert the plurality of operational data into the format by performing a plurality of data transformations on the plurality of operational data, wherein each of the plurality of data transformations receives an input data stream and produces the transformed message output.
2. The system of claim 1 , wherein the plurality of operational data is retrieved by continuous monitoring of information streams released by the plurality of network data sources.
3. The system of claim 2 , wherein at least a portion of the plurality of operational data may be isolated using filters.
4. The system of claim 1 , wherein at least a portion of the plurality of operational data is retrieved from the plurality of network data sources based upon an event trigger.
5. The system of claim 1 , wherein at least a portion of the plurality of operational data is retrieved from the plurality of network data sources based upon a time-dependent trigger.
6. The system of claim 1 , wherein at least a portion of the plurality of operational data is displayed and discarded.
7. The system of claim 1 , wherein at least a portion of the plurality of operational data is persistently stored.
8. A method for highly scalable distributed connection interface for data capture from multiple network service sources comprising the stops of:
retrieving a plurality of operational data from a plurality of network data sources;
accepting a plurality of analysis parameters and control commands from a user, the plurality of analysis parameters and control commands comprising instructions to either passively monitor data or actively retrieve the plurality of operational data from each of the plurality of network data sources, depending on the each of the plurality of network data sources;
converting, using a decomposable transformer service module of a computing device, the plurality of operational data into a format by performing a plurality of transformations on the plurality of operational data, wherein each of the plurality of transformations receives an input data stream and produces a transformed message output; and
updating a directed computational graph with the converted plurality of operational data, where the directed computational graph comprises nodes which represent a plurality of data transformations and edges which represent a plurality of transformed message outputs.
9. The method of claim 8 , wherein the plurality of operational data is retrieved by continuous monitoring of information streams released by the plurality of network data sources.
10. The method of claim 8 , wherein at least a portion of the plurality of operational data may be isolated using filters.
11. The method of claim 8 , wherein at least a portion of the plurality of operational data is retrieved from the plurality of network data sources based upon an event trigger.
12. The method of claim 8 , wherein at least a portion of the plurality of operational data is retrieved from the plurality of network data sources based upon a time-dependent trigger.
13. The method of claim 8 , wherein at least a portion of the plurality of operational data is displayed and discarded.
14. The method of claim 8 , wherein at least a portion of the plurality of operational data is persistently stored.