Method and system for transforming input data streams
View Patent ↗A system and method for processing an input data stream in a first data format of a plurality of first data formats to an output data stream in a second data format of a plurality of second data formats. A plurality of input connector modules receive respective input data streams and at least one input queue stores the received input data streams. A plurality of job threads is operatively connected to the at least one input queue, each job thread formatting a stored input data stream to produce an output data stream. At least one output queue stores the output data streams from the plurality of job threads. A plurality of output connector modules is operatively connected to the at least one output queue, the output connector modules supplying respective output data streams.
1. A method for processing a data stream in a network environment, comprising:
at a server computer, creating a plurality of input threads, wherein each input thread of the plurality of input threads listens to a physical port from which the input thread receives data;
receiving an input data stream of file data at a physical input associated with a first input thread of the plurality of input threads;
producing filtered data from the input data stream by passing the input data stream through a filter associated with the first input thread;
identifying, by an agent associated with the first input thread, each event in the filtered data, wherein the agent creates a message for each event containing text from the event and a generic data structure corresponding to the event for a thread job manager associated with the first input thread;
creating, by the thread job manager associated with the first input thread, a process for transforming the messages according to the generic data structure into output data, wherein
input data in the first data stream is of a first document format and the output data is of a second document format;
creating an output pipeline for the process; and
executing the process to produce a physical output object through the output pipeline.
2. The method according to claim 1 , wherein the output pipeline produces an output data stream, wherein the method further comprises storing an output data stream in an output queue.
3. The method according to claim 1 , further comprising:
creating a parsing model utilizing information from configuration files read by the server computer at startup.
4. The method according to claim 3 , wherein the parsing model communicates with a runtime model residing at the server computer, and wherein the runtime model contains startup components created at startup and dynamic components created during runtime.
5. The method according to claim 3 , wherein the thread job manager associated with the first input thread creates the process utilizing information in the parsing model and the message.
6. The method according to claim 1 , wherein the agent is further configured to create a field list for the event.
7. The method according to claim 1 , wherein the output pipeline is operatively connected to a receiving device via a physical output associated with the output pipeline.
8. A computer program product comprising at least one non-transitory computer readable medium storing instructions translatable by at least one processor to perform:
creating a plurality of input threads at a server computer, wherein each input thread, of the plurality of input threads listens to a physical port from which the input thread receives data;
receiving an input data stream of file data at a physical input associated with a first input thread of the plurality of input threads;
producing filtered data from the input data stream by passing the input data stream through a filter associated with the first input thread;
identifying, by an agent associated with the first input thread, each event in the filtered data, wherein the agent creates a message for each event containing text from the event and having a generic data structure corresponding to the event for a thread job manager associated with the first input thread;
creating, by the thread job manager associated with the first input thread, a process for transforming the messages according to the generic data structure into output data, wherein input data in the first data stream is of a first document format and the output data is of a second document format;
creating an output pipeline for the process; and
executing the process to produce a physical output object through the output pipeline.
9. The computer program product of claim 8 , the output pipeline includes producing an output data stream, wherein the instructions are further translatable by the at least one processor to perform: storing an output data stream in an output queue.
10. The computer program product of claim 8 , wherein the instructions are further translatable by the at least one processor to perform:
creating a parsing model utilizing information from configuration files read by the server computer at startup.
11. The computer program product of claim 10 , wherein the parsing model communicates with a runtime model residing at the server computer, and wherein the runtime model contains startup components created at startup and dynamic components created during runtime.
12. The computer program product of claim 10 , wherein the thread job manager associated with the first input thread creates the process utilizing information in the parsing model and the message.
13. The computer program product of claim 8 , wherein the agent is further configured to create a field list for the event.
14. A system for processing a data stream in a network environment, comprising:
a server computer having at least one processor; and
at least one non-transitory computer readable medium storing instructions translatable by the at least one processor to perform:
creating a plurality of input threads, wherein each input thread of the plurality of input threads listens to a physical port from which the input thread receives data;
receiving an input data stream of file data at a physical input associated with a first input thread of the plurality of input threads;
producing filtered data from the input data stream by passing the input data stream through a filter associated with the first input thread;
identifying, by an agent associated with the first input thread, each event in the filtered data, wherein the agent creates a message for each event containing text from the event and having a generic data structure corresponding to the event for a thread job manager associated with the first input thread;
creating, by the thread job manager associated with the first input thread, a process for transforming the messages according to the generic data structure into output data, wherein input data in the first data stream is of a first document format and the output data is of a second document format;
creating an output pipeline for the process; and
executing the process to produce a physical output object through the output pipeline.
15. The system of claim 14 , wherein the output pipeline produces an output data stream, wherein the instructions are further translatable by the at least one processor to perform: storing an output data stream in an output queue.
16. The system of claim 14 , wherein the instructions are further translatable by the at least one processor to perform:
creating a parsing model utilizing information from configuration files read by the server computer at startup.
17. The system of claim 16 , wherein the parsing model communicates with a runtime model residing at the server computer, and wherein the runtime model contains startup components created at startup and dynamic components created during runtime.
18. The system of claim 16 , wherein the thread job manager associated with the first input thread creates the process utilizing information in the parsing model and the message.
19. The system of claim 14 , wherein the agent is further configured to create a field list for the event.
20. The system of claim 14 , wherein the output pipeline is operatively connected to a receiving device via a physical output associated with the output pipeline.