Highly scalable distributed connection interface for data capture from multiple network service sources
A highly scalable distributed connection interface for data capture from multiple network service sources, comprising a connector module wherein, the connector module retrieves a plurality of business related data from a plurality of network data sources; employs a plurality of application programming interface routines to communicate with the plurality of business related data sources; accepts a plurality of analysis parameters and control commands directly from human interface devices or from one or more command and control storage devices; and specifies the action or actions to be taken on the retrieved business data.
1. A system for highly scalable distributed connection interface for data capture from multiple network service sources comprising:
a computing device comprising a memory, a processor, and a plurality of programming instructions:
a connector module comprising connections to a plurality of network data sources and a portion of the plurality of programming instructions configured to cause the computing device to:
employ a plurality of application programming interface routines to communicate with the plurality of network data sources;
retrieve a plurality of business related data from the plurality of network data sources;
accept a plurality of analysis parameters and control commands from a user of the system, the plurality of analysis parameters and control commands comprising:
instructions to either passively monitor data or actively retrieve data from each network data source, depending on the network data source; and
a pre-determined purpose for the data monitored or retrieved from each network data source, depending on the network data source; and
send the data and the analysis parameters and control commands to a directed computational graph module; and
a directed computational graph module comprising a directed computational graph comprising nodes representing data transformations and edges representing transformed message outputs, and a portion of the plurality of programming instructions configured to cause the computing device to:
receive the data and the analysis parameters and control commands from the connector module; and
direct a decomposable transformer service module to convert the data into a format based on the pre-determined purpose for the data; and
send the data to the decomposable transformer service module;
receive transformations and transformed message outputs from the decomposable transformer service; and
update one or more nodes of the directed computational graph to represent the transformations, and update one or more edges of the directed computational graph to represent the transformation message outputs; and
a decomposable transformer service module comprising a portion of the plurality of programming instructions configured to cause the computing device to:
receive the data and direction from the directed computational graph module;
convert the data into the format based on the pre-determined purpose for the data by performing a plurality of transformations on the data, wherein each transformation receives an input data stream and producing a transformed message output, and
send each transformation and transformed message output to the directed computational graph module.
2. The system of claim 1 , wherein the the plurality of business related data is retrieved by continuous monitoring by the application programming interface routines of information streams released by the plurality of network data sources.
3. The system of claim 2 , wherein at least a portion of the plurality of business related data may be isolated based upon use of filters.
4. The system of claim 1 , wherein at least a portion of the plurality of business related data is retrieved from the plurality of network data sources based upon an event trigger.
5. The system of claim 1 , wherein at least a portion of the plurality of business related data is retrieved from the plurality of network data sources based upon a time dependent trigger.
6. The system of claim 1 , wherein at least a portion of the business related data is transformed by the connector module into a format useful for a pre-determined purpose.
7. The system of claim 1 , wherein at least a portion of the plurality of business related data is displayed and discarded.
8. The system of claim 1 , wherein at least a portion of the plurality of business related data is persistently stored.
9. A method for highly scalable distributed connection interface for data capture from multiple network service sources comprising the steps of:
employing, within a computing device, a plurality of application programming interface routines to communicate with a plurality of network data sources;
retrieving a plurality of business related data from the plurality of network data sources;
accepting a plurality of analysis parameters and control commands from a user, the plurality of analysis parameters and control commands comprising:
instructions to either passively monitor data or actively retrieve data from each network data source, depending on the network data source; and
a pre-determined purpose for the data monitored or retrieved from each network data source, depending on the network data source; and
directing a decomposable transformer service module of the computing device to convert the data into a format based on the pre-determined purpose for the data;
converting the data into the format based on the pre-determined purpose for the data by performing a plurality of transformations on the data, wherein each transformation receives an input data stream and produces a transformed message output; and
updating a directed computational graph with the converted data, where the directed computational graph comprises nodes which represent the transformations and edges which represent the transformed message outputs.
10. The method of claim 9 , wherein the plurality of business related data is retrieved by continuous monitoring by the application programming interface routines of information streams released by the plurality of network data sources.
11. The method of claim 9 , wherein at least a portion of the plurality of business related data may be isolated based upon use of filters.
12. The method of claim 9 , wherein at least a portion of the plurality of business related data is retrieved from the plurality of network data sources based upon an event trigger.
13. The method of claim 9 , wherein at least a portion of the plurality of business related data is retrieved from the plurality of network data sources based upon a time dependent trigger.
14. The method of claim 9 , wherein at least a portion of the plurality of business related data is displayed and discarded.
15. The method of claim 9 , wherein at least a portion of the plurality of business related data is persistently stored.