Method and system for transforming input data streams
A system and method for processing an input data stream. An input connector module receives an input data streams. A job thread is operatively connected to the received input data stream and produces an output data stream. An output connector module supplies an output data stream.
1. A system for processing input data streams comprising:
a processing system coupled to an input and an output, the processing system having a non-transitory computer readable medium comprising instructions to:
read an electronic input data stream of data received at the input;
in a job thread, detect patterns in the input data stream to identify events;
in the job thread, create a message for each event, the message containing data associated with the event according to a data structure corresponding to the event;
in the job thread, execute a process configured to generate a meta record based on the message and send the meta record to an output pipeline; and
provide an output data stream to a destination from the output pipeline via the output, wherein at least a portion of the output data stream contains data from the message converted according to the meta record.
2. The system of claim 1 , wherein the output data stream comprises data not contained in the data associated with the event.
3. The system of claim 1 , wherein the processing system is coupled to the input via an input queue the processing system is coupled to the output via an output queue.
4. The system of claim 3 , wherein the input and output are physical connectors.
5. The system of claim 1 , wherein the instructions are further for splitting the data received at the input into a plurality of jobs, wherein the job thread corresponds to one of the plurality of jobs.
6. The system of claim 1 , wherein the job thread includes a plurality of job threads, each of the plurality of job threads processing a portion of the identified events.
7. A method for processing a data stream in a network environment comprising:
reading an electronic input data stream of data received at an input;
in a job thread, detecting patterns in the input data stream to identify events;
in the job thread, creating a message for each event, the message containing data associated with the event according to a data structure corresponding to the event;
in the job thread, executing a process configured to generate a meta record based on the message and send the meta record to an output pipeline; and
providing an output data stream to a destination from the output pipeline via an output, wherein at least a portion of the output data stream contains data from the message converted according to the meta record.
8. The method of claim 7 , wherein the output data stream comprises data not contained in the data associated with the event.
9. The method of claim 7 , wherein the processing system is coupled to the input via an input queue the processing system is coupled to the output via an output queue.
10. The method of claim 9 , wherein the input and output are physical connectors.
11. The method of claim 7 , wherein the instructions are further for splitting the data received at the input into a plurality of jobs, wherein the job thread corresponds to one of the plurality of jobs.
12. The method of claim 7 , wherein the job thread includes a plurality of job threads, each of the plurality of job threads processing a portion of the identified events.
13. A non-transitory computer readable medium comprising a set of computer readable instructions for:
reading an electronic input data stream of data received at an input;
in a job thread, detecting patterns in the input data stream to identify events;
in the job thread, creating a message for each event, the message containing data associated with the event according to a data structure corresponding to the event;
in the job thread, executing a process configured to generate a meta record based on the message and send the meta record to an output pipeline; and
providing an output data stream to a destination from the output pipeline via an output, wherein at least a portion of the output data stream contains data from the message converted according to the meta record.
14. The non-transitory computer readable medium of claim 13 , wherein the output data stream comprises data not contained in the data associated with the event.
15. The non-transitory computer readable medium of claim 13 , wherein the processing system is coupled to the input via an input queue the processing system is coupled to the output via an output queue.
16. The non-transitory computer readable medium of claim 15 , wherein the input and output are physical connectors.
17. The non-transitory computer readable medium of claim 13 , wherein the instructions are further for splitting the data received at the input into a plurality of jobs, wherein the job thread corresponds to one of the plurality of jobs.
18. The non-transitory computer readable medium of claim 13 , wherein the job thread includes a plurality of job threads, each of the plurality of job threads processing a portion of the identified events.