Highly scalable distributed connection interface for data capture from multiple network service sources
A highly scalable distributed connection interface for data capture from multiple network service sources, comprising a connector module wherein, the connector module retrieves a plurality of data from a plurality of network data sources; employs a plurality of application programming interface routines to communicate with the plurality of data sources; accepts a plurality of analysis parameters and control commands directly from human interface devices or from one or more command and control storage devices; and specifies the action or actions to be taken on the retrieved data.
1. A system for highly-scalable distributed connection interface for data capture from multiple network service sources comprising:
a connector module comprising connections to a plurality of network data sources and a plurality of programming instructions configured to operate on a processor of a first computing device causing the first computing device to:
retrieve a plurality of data from each of the plurality of network data sources;
accept a plurality of analysis parameters and control commands from a user of the system, the plurality of analysis parameters and control commands comprising:
instructions to either passively monitor data or actively retrieve the plurality of data from each of the plurality of network data sources, depending on the each of the plurality network data sources; and
send the plurality of data and the plurality of analysis parameters and control commands to a directed computational graph module;
the directed computational graph module comprising a directed computational graph comprising nodes representing data transformations and edges representing transformed message outputs, and a plurality of programming instructions configured to operate on a processor of a second computing device causing the second computing device to:
receive the plurality of data and the plurality of analysis parameters and control commands from the connector module;
direct a decomposable transformer service module to convert the plurality of data into a format;
receive data transformations and transformed message outputs from the decomposable transformer service module; and
update the nodes of the directed computational graph to represent the received data transformations, and update the edges of the directed computational graph to represent the received transformed message outputs; and
the decomposable transformer service module comprising a plurality of programming instructions configured to operate on a processor of a third computing device causing the third computing device to:
receive the plurality of data and direction from the directed computational graph module;
perform a plurality of transformations on the data, wherein each of the plurality of transformations receives an input data stream and produces the transformed message output; and
send each data transformation and transformed message output to the directed computational graph module.
2. The system of claim 1 , wherein the plurality of data is retrieved by continuous monitoring by application programming interface routines of information streams released by the plurality of network data sources.
3. The system of claim 2 , wherein at least a portion of the plurality of data may be isolated based upon use of filters.
4. The system of claim 1 , wherein at least a portion of the plurality of data is retrieved from the plurality of network data sources based upon an event trigger.
5. The system of claim 1 , wherein at least a portion of the plurality of data is retrieved from the plurality of network data sources based upon a time-dependent trigger.
6. The system of claim 1 , wherein at least a portion of the data is transformed by the connector module into a format useful for a predetermined purpose.
7. The system of claim 1 , wherein at least a portion of the plurality of data is displayed and discarded.
8. The system of claim 1 , wherein at least a portion of the plurality of data is persistently stored.
9. A method for highly scalable distributed connection interface for data capture from multiple network service sources comprising the steps of:
retrieving a plurality of data from a plurality of network data sources;
accepting a plurality of analysis parameters and control commands from a user, the plurality of analysis parameters and control commands comprising instructions to either passively monitor data or actively retrieve the plurality of data from each of the plurality of network data sources, depending on the each of the plurality of network data sources;
directing a decomposable transformer service module of a computing device to convert the plurality of data into a format;
converting the plurality of data into the format by performing a plurality of transformations on the plurality of data, wherein each of the plurality of transformations receives an input data stream and produces a transformed message output; and
updating a directed computational graph with the converted data, where the directed computational graph comprises nodes which represent the plurality of transformations and edges which represent the transformed message outputs.
10. The method of claim 9 , wherein the plurality of data is retrieved by continuous monitoring by application programming interface routines of information streams released by the plurality of network data sources.
11. The method of claim 9 , wherein at least a portion of the plurality of data may be isolated based upon use of filters.
12. The method of claim 9 , wherein at least a portion of the plurality of data is retrieved from the plurality of network data sources based upon an event trigger.
13. The method of claim 9 , wherein at least a portion of the plurality of data is retrieved from the plurality of network data sources based upon a time dependent trigger.
14. The method of claim 9 , wherein at least a portion of the plurality of data is displayed and discarded.
15. The method of claim 9 , wherein at least a portion of the plurality of data is persistently stored.