Method and system for transforming input data streams
A system and method for processing an input data stream. An input connector module receives an input data streams. A job thread is operatively connected to the received input data stream and produces an output data stream. An output connector module supplies an output data stream.
1. A system for processing input data streams comprising:
an input connector;
an output connector;
a processing system coupled to the input connector and the output connector, the processing system having a non-transitory computer readable medium comprising instructions to:
read an electronic input data stream of data received at the input connector;
in a job thread, detect patterns in the input data stream to identify events;
in the same job thread, create a message for each event, the message containing data associated with the event according to a generic data structure corresponding to the event;
in the same job thread, execute a process configured to generate a meta record based on the message and send the meta record to an output pipeline; and
provide an output data stream to a destination from the output pipeline via the output connector, wherein at least a portion of the output data stream contains data from the message converted according to the meta record.
2. The system of claim 1 , wherein the output data stream comprises data not contained in the data associated with the event.
3. The system of claim 1 ; wherein the processing system is coupled to the input connector via an input queue the processing system is coupled to the output connector via an output queue.
4. The system of claim 3 , wherein the input connector and output connector are physical connectors.
5. The system of claim 1 , wherein the instructions are further for splitting the data received at the input connector into a plurality of jobs, wherein the job thread corresponds to one of the plurality of jobs.
6. The system of claim 1 , wherein the job thread includes a plurality of job threads, each of the plurality of job threads processing a portion of the identified events.
7. A method for processing a data stream in a network environment comprising:
reading an electronic input data stream of data received at an input connector;
in a job thread on a process system, detecting patterns in the input data stream to identify events;
in the same job thread, creating a message for each event, the message containing data associated with the event according to a generic data structure corresponding to the event;
in the same job thread, executing a process configured to generate a meta record based on the message and send the meta record to an output pipeline; and
providing an output data stream to a destination from the output pipeline via an output connector, wherein at least a portion of the output data stream contains data from the message converted according to the meta record.
8. The method of claim 7 , wherein the output data stream comprises data not contained in the data associated with the event.
9. The method of claim 7 , wherein the processing system is coupled to the input connector via an input queue, and the processing system is coupled to the output connector via an output queue.
10. The method of claim 9 , wherein the input connector and output connector are physical connectors.
11. The method of claim 7 , wherein the method further comprises splitting the data received at the input connector into a plurality of jobs, wherein the job thread corresponds to one of the plurality of jobs.
12. The method of claim 7 , wherein the job thread includes a plurality of job threads, each of the plurality of job threads processing a portion of the identified events.
13. A non-transitory computer readable medium comprising a set of computer readable instructions for:
reading an electronic input data stream of data received at an input connector;
in a job thread on a processing system, detecting patterns in the input data stream to identify events; in the same job thread, creating a message for each event, the message containing data associated with the event according to a generic data structure corresponding to the event;
in the same job thread, executing a process configured to generate a meta record based on the message and send the meta record to an output pipeline; and
providing an output data stream to a destination from the output pipeline via an output connector, wherein at least a portion of the output data stream contains data from the message converted according to the meta record.
14. The non-transitory computer readable medium of claim 13 , wherein the output data stream comprises data not contained in the data associated with the event.
15. The non-transitory computer readable medium of claim 13 , wherein the processing system is coupled to the input connector via an input queue, and the processing system is coupled to the output connector via an output queue.
16. The non-transitory computer readable medium of claim 15 , wherein the input connector and output connector are physical connectors.
17. The non-transitory computer readable medium of claim 13 , wherein the instructions are further for splitting the data received at the input connector into a plurality of jobs, wherein the job thread corresponds to one of the plurality of jobs.
18. The non-transitory computer readable medium of claim 13 , wherein the job thread includes a plurality of job threads, each of the plurality of job threads processing a portion of the identified events.