Real time searching and reporting
A system arranged to search machine data to generate reports in real time. A search query is provided that includes a plurality of search commands. The search query is parsed to form a main search query and a remote search query. Machine data is collected from remote data sources and evaluated against one of the main and remote search queries to generate a set of search results. The main search query is then evaluated against at least a partial set of the search result to generate at least one report regarding the collected machine data. Initially a search window is pre-populated with historical machine data related to the search query. Over time the historical machine data is replaced with the collected machine data.
1. A computer-implemented method, comprising:
receiving raw event data, wherein the raw event data is received on a computing device;
dividing the raw event data into one or more events, wherein dividing includes analyzing the raw event data and generating one or more rules for establishing boundaries between events;
associating a time stamp with each event;
indexing each event using the time stamps;
storing the indexed events in an event data store;
receiving a search query;
generating a data structure, wherein the data structure is generated on the computing device, and wherein the data structure is populated by evaluating the search query against both events in the data store and raw event data being received in real-time;
generating search results by draining the data structure using the search query; and
generating a report using the search results.
2. The method of claim 1 , wherein indexing includes generating and storing index metadata.
3. The method of claim 1 , wherein the raw event data is continuously received in real-time.
4. The method of claim 1 , wherein the data structure is a queue.
5. The method of claim 1 , wherein the search query is a hybrid search query.
6. The method of claim 1 , wherein the search query is a hybrid search query, and wherein the hybrid search query is divided into a main query and a remote query, and wherein the report is generated by applying the main query to the search results.
7. The method of claim 1 , wherein the raw event data is received from one or more remote sources.
8. The method of claim 1 , wherein receiving raw event data and generating the report are done concurrently.
9. The method of claim 1 , wherein the search query is a real-time search query, and wherein the report is updated as raw event data is received.
10. The method of claim 1 , wherein the report is generated using events in the data store, and wherein the report is updated as raw event data is received.
11. A computer-implemented system, comprising:
one or more processors;
one or more non-transitory computer-readable storage mediums containing instructions configured to cause the one or more processors to perform operations including:
receiving raw event data;
dividing the raw event data into one or more events, wherein dividing includes analyzing the raw event data and generating one or more rules for establishing boundaries between events;
associating a time stamp with each event;
indexing each event using the time stamps;
storing the indexed events in an event data store;
receiving a search query;
generating a data structure, wherein the data structure is generated on the computing device, and wherein the data structure is populated by evaluating the search query against both events in the data store and raw event data being received in real-time;
generating search results by draining the data structure using the search query; and
generating a report using the search results.
12. The system of claim 11 , wherein indexing includes generating and storing index metadata.
13. The system of claim 11 , wherein the raw event data is continuously received in real-time.
14. The system of claim 11 , wherein the data structure is a queue.
15. The system of claim 11 , wherein the search query is a hybrid search query.
16. The system of claim 11 , wherein the search query is a hybrid search query, and wherein the hybrid search query is divided into a main query and a remote query, and wherein the report is generated by applying the main query to the search results.
17. The system of claim 11 , wherein the raw event data is received from one or more remote sources.
18. The system of claim 11 , wherein receiving raw event data and generating the report are done concurrently.
19. The system of claim 11 , wherein the search query is a real-time search query, and wherein the report is updated as raw event data is received.
20. The system of claim 11 , wherein the report is generated using events in the data store, and wherein the report is updated as raw event data is received.
21. A computer-program product, tangibly embodied in a non-transitory machine-readable storage medium, including instructions configured to cause a data processing apparatus to:
receive raw event data;
divide the raw event data into one or more events, wherein dividing includes analyzing the raw event data and generating one or more rules for establishing boundaries between events;
associate a time stamp with each event;
index each event using the time stamps;
store the indexed events in an event data store;
receive a search query;
generate a data structure, wherein the data structure is generated on the computing device, and wherein the data structure is populated by evaluating the search query against both events in the data store and raw event data being received in real-time;
generate search results by draining the data structure using the search query; and
generate a report using the search results.
22. The computer-program product of claim 21 , wherein indexing includes generating and storing index metadata.
23. The computer-program product of claim 21 , wherein the raw event data is continuously received in real-time.
24. The computer-program product of claim 21 , wherein the data structure is a queue.
25. The computer-program product of claim 21 , wherein the search query is a hybrid search query.
26. The computer-program product of claim 21 , wherein the search query is a hybrid search query, and wherein the hybrid search query is divided into a main query and a remote query, and wherein the report is generated by applying the main query to the search results.
27. The computer-program product of claim 21 , wherein the raw event data is received from one or more remote sources.
28. The computer-program product of claim 21 , wherein receiving raw event data and generating the report are done concurrently.
29. The computer-program product of claim 21 , wherein the search query is a real-time search query, and wherein the report is updated as raw event data is received.
30. The computer-program product of claim 21 , wherein the report is generated using events in the data store, and wherein the report is updated as raw event data is received.