Streaming data processing
Systems and methods are disclosed for processing streaming data. The data can come from various sources. Worker nodes can be configured to process the streaming data, without delays that may be caused by indexing the data. The data can be filtered and/or transformed as it is processed. In some cases, data can be stored in a data store without transformation. The data in the data store can be accessed and processed at a later time.
1. A system, comprising:
non-transitory computer-readable media including computer-executable instructions; and
a processor configured to execute the computer-executable instructions, wherein execution of the computer-executable instructions causes the system to:
responsive to receiving a portion of a stream of unindexed machine generated data containing first unindexed events, each unindexed event of the first unindexed events including a portion of machine data of the unindexed machine generated data:
communicate a copy of the first unindexed events included within the received portion of the stream to a data store without transforming the first unindexed events;
apply at least one filter to the first unindexed events to provide first filtered unindexed events;
transform at least a portion of data of at least one filtered unindexed event of the first filtered unindexed events to provide at least one transformed unindexed event; and
communicate the at least one transformed unindexed event to a data destination that is different from the data store;
receive second unindexed events as part of the stream of unindexed machine generated data, wherein each event of the second unindexed events includes a portion of machine data from the stream of unindexed machine generated data;
store the second unindexed events in the data store without transforming the second unindexed events;
at a later time, retrieve the second unindexed events from the data store;
apply at least one filter to the second unindexed events to provide second filtered unindexed events;
transform at least a portion of data of the second filtered unindexed events to provide at least one second transformed unindexed event; and
communicate the at least one second transformed unindexed event to the data destination.
2. The system of claim 1 , wherein the stream of unindexed machine generated data is a continuous stream of unindexed machine generated data.
3. The system of claim 1 , wherein the processor is configured to receive the portion of the stream of unindexed machine generated data from a message queue of a pub-sub.
4. The system of claim 1 , wherein receiving the portion of the stream of unindexed machine generated data comprises receiving the unindexed events from a topic of a pub-sub.
5. The system of claim 1 , wherein the processor is configured to receive the portion of the stream of unindexed machine generated data from a computing device configured to generate the portion of the stream of unindexed machine generated data from the stream of unindexed machine generated data by breaking the stream of unindexed machine generated data into respective portions.
6. The system of claim 1 , wherein to apply at least one filter to the first unindexed events, the processor is configured to filter the first unindexed events based on metadata associated with the first unindexed events.
7. The system of claim 1 , wherein to apply at least one filter to the first unindexed events, the processor is configured to filter the first unindexed events based on a sourcetype of the first unindexed events.
8. The system of claim 1 , wherein to transform at least a portion of data of the at least one filtered unindexed event, the processor is configured to transform at least a portion of data of the at least one filtered unindexed event based on task instructions of a data pipeline.
9. The system of claim 1 , wherein to transform at least a portion of data of the at least one filtered unindexed event, the processor is configured to annotate the at least one filtered unindexed event based on task instructions of a data pipeline.
10. The system of claim 1 , wherein to transform at least a portion of data of the at least one filtered unindexed event, the processor is configured to mask at least a portion of data of the at least one filtered unindexed event based on task instructions of a data pipeline.
11. The system of claim 1 , wherein to transform at least a portion of data of the at least one filtered unindexed event, the processor is configured to remove at least a portion of data of the at least one filtered unindexed event based on task instructions of a data pipeline.
12. The system of claim 1 , wherein to communicate the at least one transformed unindexed event to a data destination, the processor is configured to communicate the at least one transformed unindexed event to a queue, wherein one or more data indexing devices retrieve the at least one transformed unindexed event from the queue, index the at least one transformed unindexed event to provide at least one indexed event, and store the at least one indexed event in a time series data store.
13. The system of claim 1 , wherein to communicate the at least one transformed unindexed event to a data destination, the processor is configured to communicate the at least one transformed unindexed event to one or more data indexing devices, wherein the one or more data indexing devices index the at least one transformed unindexed event to provide at least one indexed event, and store the at least one indexed event in a time series data store.
14. The system of claim 1 , wherein the processor is further configured to communicate the at least one transformed unindexed event to the data store.
15. The system of claim 1 , wherein the processor is further configured to:
at a later time, retrieve at least one unindexed event of the copy of the first unindexed events from the data store;
transform at least a portion of data of the at least one unindexed event to provide at least one second transformed unindexed event; and
communicate the at least one second transformed unindexed event to the data destination.
16. The system of claim 1 , wherein the processor is further configured to:
at a later time, retrieve at least one unindexed event of the copy of the first unindexed events from the data store;
transform at least a portion of data of the at least one unindexed event to provide at least one second transformed unindexed event; and
communicate the at least one second transformed unindexed event to the data store.
17. The system of claim 1 , wherein the processor is further configured to:
at a later time, apply at least one filter to the copy of the first unindexed events to retrieve at least one unindexed event of the copy of the first unindexed events from the data store;
transform at least a portion of data of the at least one unindexed event to provide at least one second transformed unindexed event; and
communicate the at least one second transformed unindexed event to the data destination.
18. The system of claim 1 , wherein the data destination is a queue, and wherein one or more data indexing devices retrieve the at least one transformed unindexed event from the queue, index the at least one transformed unindexed event to provide at least one indexed event, and store the at least one indexed event in a time series data store, wherein the processor is further configured to:
at a later time, retrieve at least one unindexed event of the copy of the first unindexed events from the data store;
transform at least a portion of data of the at least one unindexed event to provide at least one second transformed unindexed event; and
communicate the at least one second transformed unindexed event to the queue, wherein one or more data indexing devices retrieve the at least one second transformed unindexed event from the queue, index the unindexed events, and store indexed events in a time series data store.
19. The system of claim 1 , wherein the processor is further configured to:
at a later time, retrieve at least one unindexed event of the copy of the first unindexed events from the data store;
transform at least a portion of data of the at least one unindexed event to provide at least one second transformed unindexed event; and
communicate the at least one second transformed unindexed event to the data store.
20. A computer-implemented method, comprising:
responsive to receiving a portion of a stream of unindexed machine generated data containing first unindexed events, each unindexed event of the first unindexed events including a portion of machine data of the unindexed machine generated data:
communicating a copy, of the first unindexed events included within the received portion of the stream, to a data store without transforming the unindexed events;
applying at least one filter to the first unindexed events to provide filtered unindexed events;
transforming at least a portion of data of the first filtered unindexed events to provide at least one transformed unindexed event; and
communicating the at least one transformed unindexed event to a data destination that is different from the data store;
receiving second unindexed events as part of the stream of unindexed machine generated data, wherein each event of the second unindexed events includes a portion of machine data from the stream of unindexed machine generated data;
storing the second unindexed events in the data store without transforming the second unindexed events;
at a later time, retrieving the second unindexed events from the data store;
applying at least one filter to the second unindexed events to provide second filtered unindexed events;
transforming at least a portion of data of the second filtered unindexed events to provide at least one second transformed unindexed event; and
communicating the at least one second transformed unindexed event to the data destination.
21. The computer-implemented method of claim 20 , further comprising communicating the at least one transformed unindexed event to the data store.
22. The computer-implemented method of claim 20 , further comprising:
at a later time, retrieving at least one unindexed event of the copy of the first unindexed events from the data store;
transforming at least a portion of data of the at least one unindexed event to provide at least one second transformed unindexed event; and
communicating the at least one second transformed unindexed event to the data destination.
23. The method of claim 20 , further comprising:
at a later time, retrieving at least one unindexed event of the copy of the unindexed events from the data store;
transforming at least a portion of data of the at least one unindexed event to provide at least one second transformed unindexed event; and
communicating the at least one second transformed unindexed event to the data store.
24. One or more non-transitory computer-readable media including computer-executable instructions that, when executed, cause a computing system to:
responsive receiving a portion of a stream of unindexed machine generated data containing first unindexed events, each unindexed event of the first unindexed events including a portion of machine data of the unindexed machine generated data:
communicate a copy, of the first unindexed events included within the received portion of the stream, to a data store without transforming the first unindexed events;
apply at least one filter to the first unindexed events to provide first filtered unindexed events;
transform at least a portion of data of at least one filtered unindexed event of the first filtered unindexed events to provide at least one transformed unindexed event; and
communicate the at least one transformed unindexed event to a data destination that is different from the data store;
receive second unindexed events as part of the stream of unindexed machine generated data, wherein each event of the second unindexed events includes a portion of machine data from the stream of unindexed machine generated data;
store the second unindexed events in the data store without transforming the second unindexed events;
at a later time, retrieve the second unindexed events from the data store;
apply at least one filter to the second unindexed events to provide second filtered unindexed events;
transform at least a portion of data of the second filtered unindexed events to provide at least one second transformed unindexed event; and
communicate the at least one second transformed unindexed event to the data destination.
25. The one or more non-transitory computer-readable media of claim 24 , wherein the computer-executable instructions further cause the computing system to communicate the at least one transformed unindexed event to the data store.
26. The one or more non-transitory computer-readable media of claim 24 , wherein the computer-executable instructions further cause the computing system to:
at a later time, retrieve at least one unindexed event of the copy of the first unindexed events from the data store;
transform at least a portion of data of the at least one unindexed event to provide at least one second transformed unindexed event; and
communicate the at least one second transformed unindexed event to the data destination.