Conversion of structured queries into unstructured queries for searching unstructured data store including timestamped raw machine data
Technologies are described herein for executing queries expressed with reference to a structured query language against unstructured data. A user issues a structured query through a traditional structured data management (“SDM”) application. Upon receiving the structured query, an SDM driver analyzes the structured query and extracts a data structure from the unstructured data, if necessary. The structured query is then converted to an unstructured query based on the extracted data structure. The converted unstructured query may then be executed against the unstructured data. Results from the query are reorganized into structured data utilizing the extracted data structure and are then presented to the user through the SDM application.
1. A computer-implemented method comprising:
segmenting unstructured raw machine data into a plurality of events; associating a timestamp with each event of the plurality of events;
storing the plurality of events as unstructured data in an unstructured data store with associated timestamps, wherein the unstructured data in the unstructured data store includes the unstructured raw machine data that has been segmented and timestamped;
receiving, at a query converter, a structured query in a structured query language from an application;
generating, by the query converter, a second query in a second query language associated with the unstructured data store, based on the structured query;
causing execution of the second query against the unstructured data stored in the unstructured data store;
receiving a result of execution of the second query against the unstructured data stored in the unstructured data store; and
causing an indication of the result to be provided to the application for output to a user, wherein the indication of the result is provided to the application as a direct response to the structured query, without requiring any additional query from the application.
2. The computer-implemented method of claim 1 , further comprising:
at the query converter, identifying a first set of fields in the unstructured data to obtain field identification data from the unstructured data, the unstructured data including text records, each of the fields in the first set of fields corresponding to a portion of text extracted from a portion of at least one of the text records;
wherein generating the second query in the second query language associated with the unstructured data store includes using the identified first set of fields to generate the second query.
3. The computer-implemented method of claim 2 , wherein the second query causes one or more values for one or more fields included in the second query to be extracted as a function of a format of the unstructured data.
4. The computer-implemented method of claim 2 , further comprising caching the identified first set of fields.
5. The computer-implemented method of claim 2 , wherein identifying a first set of fields performs the query on a subset of the unstructured data, and wherein the subset of the unstructured data is of a definable size.
6. The computer-implemented method of claim 2 , wherein identifying a first set of fields automatically identifies fields in the unstructured data as a function of formatting of the unstructured data.
7. The computer-implemented method of claim 2 , wherein the first query comprises a Structured Query Language (“SQL”) query.
8. The computer-implemented method of claim 1 , further comprising:
identifying a value of a field in an event stored in the unstructured data store, based on an extraction rule that specifies where to find a subportion of text within an event.
9. A computer-implemented method as recited in claim 1 , further comprising:
identifying a field in an event stored in the unstructured data store;
wherein generating the second query in the second query language associated with the unstructured data store is based on an identification of the field resulting from said identifying.
10. A computer-readable storage medium having computer-executable instructions stored thereupon which, when executed by a computer, cause the computer to:
segment unstructured raw machine data into a plurality of events;
associate a timestamp with each event of the plurality of events; store the plurality of events as unstructured data in an unstructured data store with associated timestamps, wherein the unstructured data in the unstructured data store includes the unstructured raw machine data that has been segmented and timestamped;
receive, at a query converter, a structured query in a structured query language from an application;
generate, by the query converter, a second query in a second query language associated with the unstructured data store, based on the structured query;
cause execution of the second query against the unstructured data stored in the unstructured data store;
receive a result of execution of the second query against the unstructured data stored in the unstructured data store; and
cause an indication of the result to be provided to the application for output to a user, wherein the indication of the result is provided to the application as a direct response to the structured query, without requiring any additional query from the application.
11. The computer-readable storage medium of claim 10 , wherein said instructions further comprise instructions that when executed by the processor, cause the processor to:
identify a first set of fields in the unstructured data to obtain field identification data from the unstructured data source, the unstructured data including text records, each of the fields in the first set of fields corresponding to a portion of text extracted from a portion of at least one of the text records;
wherein generating the second query in the second query language associated with the unstructured data store includes generating the second query by using the identified first set of fields.
12. The computer-readable storage medium of claim 11 , wherein the second query causes one or more values for one or more fields included in the second query to be extracted as a function of a format of the unstructured data.
13. The computer-readable storage medium of claim 11 , further comprising computer-executable instructions stored thereupon which, when executed by a computer, cause the computer to:
cache the identified first set of fields.
14. The computer-readable storage medium of claim 11 , wherein querying of the unstructured data to identify the first set of fields is performed on a subset of the unstructured data, and wherein the subset of the unstructured data is of a definable size.
15. The computer-readable storage medium of claim 11 , wherein the first set of fields is identified by automatically identifying fields in the unstructured data as a function of formatting of the unstructured data.
16. The computer-readable storage medium of claim 11 , wherein the first query comprises a Structured Query Language (“SQL”) query.
17. A system comprising:
a processor and instructions in memory coupled to the processor that, when executed by the processor, cause the system to perform operations comprising:
segmenting unstructured raw machine data into a plurality of events; associating a timestamp with each event of the plurality of events;
storing the plurality of events as unstructured data in an unstructured data store with associated timestamps, wherein the unstructured data in the unstructured data store includes the unstructured raw machine data that has been segmented and timestamped;
receiving, at a query converter, a structured query in a structured query language from an application;
generating, by the query converter, a second query in a second query language associated with the unstructured data store, based on the structured query;
causing execution of the second query against the unstructured data stored in the unstructured data store;
receiving a result of execution of the second query against the unstructured data stored in the unstructured data store; and
causing an indication of the result to be provided to the application for output to a user, wherein the indication of the result is provided to the application as a direct response to the structured query, without requiring any additional query from the application.
18. The system of claim 17 , said instructions further comprising instructions that, when executed by the processor, cause the system to perform operations comprising:
identifying a first set of fields in the unstructured data to obtain field identification data from the unstructured data source, the unstructured data including text records, each of the fields in the first set of fields corresponding to a portion of text extracted from a portion of at least one of the text records;
wherein generating the second query in the second query language associated with the unstructured data store includes generating the second query by using the identified first set of fields.
19. The system of claim 18 , wherein the second query causes one or more values for one or more fields included in the second query to be extracted as a function of a format of the unstructured data.
20. The system of claim 18 , wherein the system is further configured to cache the first set of fields.
21. The system of claim 18 , wherein the first set of fields is identified by performing the query on a subset of the unstructured data, and wherein the subset of the unstructured data is of a definable size.
22. The system of claim 18 , wherein the first set of fields is identified by automatically identifying fields in the unstructured data as a function of formatting of the unstructured data.
23. The system of claim 18 , wherein the first query comprises a Structured Query Language (“SQL”) query.