DISTRIBUTED INTERFACE FOR DATA CAPTURE FROM MULTIPLE SOURCES
A system and method for a highly scalable distributed connection interface for data capture from multiple network service sources. The connection interface is designed to enable simple to initiate, performant and highly available input/output from a large plurality of external networked service's and application's application programming interfaces (API) to the modules of an integrated predictive business operating system. To handle the high volume of information exchange, the connection interface is distributed and designed to be scalable and self-load-balancing. The connection interface possesses robust expressive scripting capabilities that allow highly specific handling rules to be generated for the routing, transformation, and output of data within the business operating system.
1 . A system for providing a distributed connection interface for data capture from cloud-based service sources comprising:
a computing device comprising a memory, a processor, and a non-volatile data storage device;
a component input server comprising a first plurality of programming instructions stored in the memory of, and operating on the processor of, the computing device, wherein the first plurality of programming instructions, when operating on the processor, cause the computing device to:
receive a connector workflow configuration from a representational state transfer (REST) server;
retrieve a plurality of data messages from a data message queue;
store a plurality of retrieved data messages in a cache;
send the connector workflow configuration to an execution server;
send a first data message of the plurality of data messages to any available execution server;
receive an acknowledge signal from the respective execution server; and
remove the first data message from the cache when the acknowledge signal is received;
the execution server comprising a second plurality of programming instructions stored in the memory of, and operating on the processor of, the computing device, wherein the second plurality of programming instructions, when operating on the processor, cause the computing device to:
receive the first data message from the component input server;
send the acknowledge signal to the component input server when the first data message has been received;
create instances of component plugins to start a connector workflow process based on the connector workflow configuration;
store a state of the connector workflow process; and
restart a connector workflow process in the event of workflow stoppage using the stored state of the connector workflow process.
2 . The system of claim 1 , further comprising:
a REST server comprising a third plurality of programming instructions stored in the memory of, and operating on the processor of, the computing device, wherein the third plurality of programming instructions, when operating on the processor, cause the computing device to:
receive the connector workflow configuration;
instantiate the component input server;
start the connector workflow process;
send the connector workflow process to the component input server;
receive stored state details from the execution server of a completed connector workflow process; and
generate a workflow report based at least in part on the received stored state details.
3 . The system of claim 1 , wherein the computing device is comprised of multiple computers forming a distributed system.
4 . The system of claim 1 , wherein the connector workflow configuration is a data processing workflow generated from a distributed computational graph module.
5 . The system of claim 1 , wherein the REST server comprises a stateless application programming interface that facilitates data exchange between and among cloud-based services.
6 . The system of claim 1 , wherein the output of the connector workflow process is used as the input to one or more additional connector workflow processes, thus facilitating the construction of complex connector workflow processes from the combination and interconnection of individual connector workflow processes.
7 . A method for providing a distributed connection interface for data capture from cloud-based service sources, comprising the steps of:
using a component input server operating on a computing device comprising a memory, a processor, and a non-volatile data storage device to perform the steps of:
receiving a connector workflow configuration from a representational state transfer (REST) server;
retrieving a plurality of data messages from a data message queue;
storing a plurality of retrieved data messages in a cache;
sending a connector workflow configuration to an execution server operating on the computer device;
sending a first data message of the plurality of data messages to any available execution server;
receiving an acknowledge signal from the execution server;
removing the first data message from the cache when the acknowledge signal is received;
using the execution server operating on the computing device to perform the steps of:
receiving the first data message from the component input server;
sending the acknowledge signal to the component input server when the first data message has been received;
creating instances of component plugins to start a connector workflow process based on the connector workflow configuration;
storing a state of the connector workflow process; and
restarting the connector workflow process in the event of workflow stoppage using the stored state of the connector workflow process.
8 . The method of claim 7 , further comprising the steps of:
using the REST server operating on the computing device to perform the steps of:
receiving the connector workflow configuration;
instantiating the component input server;
starting the connector workflow process;
sending the connector workflow configuration to the component input server;
receiving stored state details from the execution server of a completed connector workflow process; and
generating a workflow report based at least in part on the received stored state details.
9 . The method of claim 7 , wherein the computing device is comprised of multiple computers forming a distributed system.
10 . The method of claim 7 , wherein the connector workflow configuration is a data processing workflow generated from a distributed computational graph module.
11 . The method of claim 8 , wherein the REST server comprises a stateless application programming interface that facilitates data exchange between and among cloud-based services.
12 . The method of claim 7 , wherein the output of the connector workflow process is used as the input to one or more connector workflow processes, thus facilitating the construction of complex connector workflow processes from the combination and interconnection of individual connector workflow processes.