Method and system for transforming input data streams
View Patent ↗A system and method for processing an input data stream in a first data format of a plurality of first data formats to an output data stream in a second data format of a plurality of second data formats. A plurality of input connector modules receive respective input data streams and at least one input queue stores the received input data streams. A plurality of job threads is operatively connected to the at least one input queue, each job thread formatting a stored input data stream to produce an output data stream. At least one output queue stores the output data streams from the plurality of job threads. A plurality of output connector modules is operatively connected to the at least one output queue, the output connector modules supplying respective output data streams.
1. A system for transforming input data streams comprising:
a physical input connector;
a physical output connector;
a processing system coupled to the physical input connector and the physical output connector, the processing system including a computer program product to:
read an electronic input data stream of file data received at the physical input connector, wherein input data in the input data stream is of a first document format;
in a same thread:
detect patterns in an input file of the input data stream to identify events;
create a message for each identified event containing text from the event according to a generic data structure corresponding to the event;
execute a process configured to create output data of a second format from the messages, the output data created from a processed message containing text from the processed message, the output data in a different format from the first document format; and
provide an output data stream to a destination via the physical output connector, the output data stream comprising the output data.
2. The system for transforming input data streams of claim 1 , wherein the processing system creates an output pipeline for the process.
3. The system for transforming input data streams of claim 2 , wherein the output pipeline further comprises a device driver.
4. The system for transforming input data streams of claim 3 , wherein the output data created from the processed message comprises a meta record containing instructions to a device driver on how to output text from the processed message and the process sends the meta record to the output pipeline.
5. The system for transforming input data streams of claim 4 , wherein:
the output pipeline comprises a physical output object configured to deliver the output data stream to the destination; and
the device driver is configured to convert the meta record to device specific output that is provided to the physical output object.
6. The system for transforming input data streams of claim 5 , wherein the output pipeline further comprises one or more additional output objects between the process and the driver.
7. The system for transforming input data streams of claim 1 , further comprising an input queue and an output queue, where the processing system reads the input data stream from the input queue and queue the output data stream in the output queue.
8. The system for transforming input data streams of claim 1 , wherein the processing system comprises a multi-threading system.
9. A computer program product comprising a non-transitory computer readable medium storing a set of computer readable instructions, the set of computer readable instructions comprising instructions executable to:
read an electronic input data stream of file data received over a network at a physical input, wherein input data in the input data stream is of a first document format;
in a same thread:
detect patterns in an input file of the input data stream to identify events;
create a message for each identified event containing text from the event according to a generic data structure corresponding to the event;
execute a process configured to create output data of a second format from the messages, the output data created from a processed message containing text from the processed message, the output data in a different format from the first document format; and
send an output data stream to a destination, the output data stream comprising the output data.
10. The computer program product of claim 9 , further comprising instruction executable to create an output pipeline for the process.
11. The computer program product of claim 10 , wherein the output pipeline further comprises a device driver.
12. The computer program product of claim 11 , wherein the output data created from the processed message comprises a meta record containing instructions to the device driver on how to output text from the processed message and the process is configured to send the meta record to the output pipeline.
13. The computer program product of claim 12 , further comprising instructions executable to:
convert the meta record to device specific output at the device driver;
provide the device specific output to a physical output object; and
send the output data stream from the physical output object to the destination.
14. The computer program product of claim 11 , further comprising instructions executable to implement one or more additional output objects in the output pipeline between the process and the device driver.
15. A method for processing a data stream in a network environment, comprising:
receiving an electronic input data stream of file data at a physical input over a network, wherein input data in the input data stream is of a first document format;
in a same thread:
detecting patterns in an input file of the input data stream to identify events;
creating a message for each identified event containing text from the event according to a generic data structure corresponding to the event;
executing a process configured to create output data of a second format from the messages, the output data created from a processed message containing text from the processed message, the output data in a different format from the first document format; and
sending an output data stream to a destination, the output data stream comprising the output data.
16. The method of claim 15 , further comprising creating an output pipeline for the process.
17. The method of claim 16 , wherein the output pipeline further comprises a device driver.
18. The method of claim 17 , wherein the output data created from the processed message comprises a meta record containing instructions to the device driver on how to output text from the processed message and the process is configured to send the meta record to the output pipeline.
19. The method of claim 18 , further comprising:
converting the meta record to device specific output at the device driver;
providing the device specific output to a physical output object; and
sending the output data stream from the physical output object to the destination.
20. The method of claim 19 , further comprising implementing one or more additional output objects in the output pipeline between the process and the driver.
21. A system for transforming input data streams comprising:
a physical input connector;
a physical output connector;
a processing system coupled to the physical input connector and the physical output connector, the processing system including a computer program product to:
read an electronic input data stream of file data received at the physical input connector, wherein input data in the input data stream is of a first document format;
in a same thread:
detect patterns in an input file of the input data stream to identify events;
create a message for each identified event containing text from the event according to a generic data structure corresponding to the event;
execute a process configured to create output data of a second format from the messages, the output data created from a message processed according to a generic structure defining one or more fields, blocks or variables associated with events, the output data containing text from the processed message and in a different format from the first document format; and
provide an output data stream to a destination via the physical output connector, the output data stream comprising the output data.