Highly scalable distributed connection interface for data capture from multiple network service and cloud-based sources
A system and method for a highly scalable distributed connection interface for data capture from multiple network service sources. The connection interface is designed to enable simple to initiate, performant and highly available input/output from a large plurality of external networked service's and application's application programming interfaces (API) to the modules of an integrated predictive business operating system. To handle the high volume of information exchange, the connection interface is distributed and designed to be scalable and self-load-balancing. The connection interface possesses robust expressive scripting capabilities that allow highly specific handling rules to be generated for the routing, transformation, and output of data within the business operating system.
1. A system for highly scalable distributed connection interface for data capture from cloud-based service sources implemented in a clustered schema comprising:
a computing device comprising a memory, a processor, and a non-volatile data storage device;
a component input server comprising a first plurality of programming instructions stored in the memory of, and operating on the processor of, the computing device, wherein the first plurality of programming instructions, when operating on the processor, cause the computing device to:
receive a connector workflow configuration from a representational state transfer (REST) server;
retrieve a plurality of data messages from a data message queue;
store the plurality of retrieved data messages in a cache;
send the connector workflow configuration to an execution server;
send a first data message of the plurality of data messages to any available execution server;
receive an acknowledgement signal from the execution server; and
remove the first data message from the cache when the acknowledge signal is received;
the execution server comprising a second plurality of programming instructions stored in the memory of, and operating on the processor of, the computing device, wherein the second plurality of programming instructions, when operating on the processor, cause the computing device to:
receive the first data message from the component input server;
send the acknowledge signal to the component input server when a the first data message has been received;
create instances of component plugins to start a connector workflow process based on the connector workflow configuration;
store a state of the connector workflow process;
restart a connector workflow process in the event of workflow stoppage using the stored state of the connector workflow process; and
send an output of the connector workflow process when completed and the stored state of the connector workflow process to a the REST server; and
the REST server comprising a third plurality of programming instructions stored in the memory of, and operating on the processor of, the computing device, wherein the third plurality of programming instructions, when operating on the processor, cause the computing device to:
receive the connector workflow configuration;
instantiate the component input server;
start the connector workflow process;
send the connector workflow process to the component input server;
receive the stored state details from the execution server;
receive the output of the connector workflow; and
generate a workflow report from the output.
2. The system of claim 1 , wherein the computing device is comprised of multiple computers forming a distributed system.
3. The system of claim 1 , wherein the connector workflow configuration is a data processing workflow generated from a distributed computational graph module.
4. The system of claim 1 , wherein the REST server is a RESTful application programming interface (API) that facilitates data exchange between and among cloud-based services.
5. The system of claim 1 , wherein the output of the connector workflow process is used as the input to one or more additional connector workflow processes, thus facilitating the construction of complex connector workflow processes from the combination and interconnection of individual connector workflow processes.
6. A method for highly scalable distributed connection interface for data capture from multiple network service sources implemented in a clustered schema, comprising the steps of:
using a component input server operating on a computing device comprising a memory, a processor, and a non-volatile data storage device to perform the steps of:
receiving a connector workflow configuration from a representational state transfer (REST) server operating on the computing device;
retrieving a plurality of data messages from a data message queue;
storing the plurality of data messages in a cache;
sending a connector workflow configuration to an execution server operating on the computing device;
sending a first data message of the plurality of data messages to any available execution server;
receiving an acknowledge signal from the execution server;
removing the first data message from the cache when the acknowledge signal is received;
using the execution server operating on the computing device to perform the steps of:
receiving the first data message from the component input server;
sending the acknowledge signal to the component input server when a the first data message has been received;
create instances of component plugins to start a connector workflow process based on the connector workflow configuration;
storing a state of the connector workflow process;
restarting the connector workflow process in the event of workflow stoppage using the stored state of the connector workflow process; and
sending an output of the connector workflow process when completed and current execution state details to the REST server; and
using the REST server operating on the computing device to perform the steps of:
receiving the connector workflow configuration;
instantiating the component input server;
starting the connector workflow process;
sending the connector workflow configuration to the component input server;
receiving the stored state of the connector workflow process from the execution server;
receiving the output of a connector workflow; and
generating a workflow report from the output.
7. The method of claim 6 , wherein the computing device is comprised of multiple computers forming a distributed system.
8. The method of claim 6 , wherein the connector workflow configuration is a data processing workflow generated from a distributed computational graph module.
9. The method of claim 6 , wherein the REST server is a RESTful application programming interface (API) that facilitates data exchange between and among cloud-based services.
10. The method of claim 6 , wherein the output of the connector workflow process is used as the input to one or more additional connector workflow processes, thus facilitating the construction of complex connector workflow processes from the combination and interconnection of individual connector workflow processes.