Hybrid structured/unstructured search and query system
Technologies are described herein for executing queries expressed with reference to a structured query language against unstructured data. A user issues a structured query through a traditional structured data management (“SDM”) application. Upon receiving the structured query, an SDM driver analyzes the structured query and extracts a data structure from the unstructured data, if necessary. The structured query is then converted to an unstructured query based on the extracted data structure. The converted unstructured query may then be executed against the unstructured data. Results from the query are reorganized into structured data utilizing the extracted data structure and are then presented to the user through the SDM application.
1. A computer-implemented method comprising:
obtaining, by a query converter, a structured query generated by an application in a structured query language, the structured query representing a requested search of raw machine data, the raw machine data being stored in an unstructured data store as unstructured raw machine data that has been segmented and timestamped;
identifying, by the query converter, a first set of fields in the unstructured data to obtain field identification data from the unstructured data, the unstructured data including records, each of the fields in the first set of fields corresponding to a portion of information extracted from a portion of at least one of the records;
generating, by the query converter, an unstructured query in a second query language associated with the unstructured data store, based on the structured query, wherein generating the unstructured query in the second query language includes using the identified first set of fields to generate the unstructured query;
causing execution of the unstructured query against the unstructured data stored in the unstructured data store;
receiving a result of execution of the unstructured query against the unstructured data stored in the unstructured data store; and
causing an indication of the result to be provided to the application, wherein the indication of the result is provided to the application as a direct response to the structured query without requiring any additional query from the application.
2. The computer-implemented method of claim 1 , wherein the unstructured raw machine data has been segmented into a plurality of events prior to said receiving the structured query;
the method further comprising:
identifying a value of a field in an event stored in the unstructured data store, based on an extraction rule that specifies where to find a subportion of text within an event.
3. The computer-implemented method as recited in claim 1 , wherein the unstructured raw machine data has been segmented into a plurality of events prior to said obtaining the structured query;
the method further comprising:
identifying a field in an event stored in the unstructured data store;
wherein generating the unstructured query in the second query language associated with the unstructured data store is based on an identification of the field.
4. The computer-implemented method of claim 1 , wherein the unstructured query causes one or more values for one or more fields included in the unstructured query to be extracted as a function of a format of the unstructured data.
5. The computer-implemented method as recited in claim 1 , further comprising:
applying a schema to the raw machine data in the unstructured data store after the raw machine data has been stored in the unstructured data store, to impose a structure on the raw machine data.
6. The computer-implemented method of claim 1 , wherein said identifying the first set of fields includes performing the query on a subset of the unstructured raw machine data, and wherein the subset of the unstructured data is of a definable size.
7. The computer-implemented method of claim 1 , wherein said identifying the first set of fields includes automatically identifying fields in the unstructured data as a function of formatting of the unstructured data.
8. The computer-implemented method of claim 1 , wherein the structured query comprises a Structured Query Language (“SQL”) query.
9. The computer-implemented method as recited in claim 1 , wherein at least some of the unstructured data is stored in the unstructured data store in a JSON format or a JSON-based format.
10. The computer-implemented method as recited in claim 1 , further comprising:
in response to receiving the result of execution of the unstructured query, converting the result into a format compatible with an output capability of the application, prior to causing the indication of the result to be provided to the application for output to a user.
11. The computer-implemented method as recited in claim 1 , wherein the raw machine data includes data indicative of performance or operation of one or more components of an information technology environment.
12. The computer-implemented method as recited in claim 1 , further comprising:
transmitting a pilot query to the unstructured data store;
responsive to the pilot query, receiving a plurality of records generated by execution of the pilot query against the unstructured raw machine data in the unstructured data store, each record including one or more pairs of field names and values;
automatically identifying field names and data types from the pairs in the plurality of records generated by execution of the pilot query against the unstructured data store; and
defining a first set of fields that corresponds to the identified field names and data types.
13. A non-transitory machine-readable storage medium having computer-executable instructions stored therein which, when executed by a computer, cause the computer to perform operations comprising:
obtain a structured query generated by an application in a structured query language, the structured query representing a requested search of raw machine data, the raw machine data being stored in an unstructured data store as unstructured raw machine data that has been segmented and timestamped;
identify a first set of fields in the unstructured data to obtain field identification data from the unstructured data, the unstructured data including records, each of the fields in the first set of fields corresponding to a portion of information extracted from a portion of at least one of the records;
generate an unstructured query in a second query language associated with the unstructured data store, based on the structured query, wherein generating the unstructured query in the second query language includes using the identified first set of fields to generate the unstructured query;
cause execution of the unstructured query against the unstructured data stored in the unstructured data store;
receive a result of execution of the unstructured query against the unstructured data stored in the unstructured data store; and
cause an indication of the result to be provided to the application, wherein the indication of the result is provided to the application as a direct response to the structured query without requiring any additional query from the application.
14. The non-transitory machine-readable storage medium of claim 13 , wherein the unstructured raw machine data has been segmented into a plurality of events prior to said receiving the structured query;
said operations further comprising:
identifying a value of a field in an event stored in the unstructured data store, based on an extraction rule that specifies where to find a subportion of text within an event.
15. The non-transitory machine-readable storage medium of claim 13 , wherein the unstructured raw machine data has been segmented into a plurality of events prior to said obtaining the structured query;
said operations further comprising:
identifying a field in an event stored in the unstructured data store;
wherein generating the unstructured query in the second query language associated with the unstructured data store is based on an identification of the field.
16. The non-transitory machine-readable storage medium of claim 13 , said operations further comprising:
applying a schema to the raw machine data in the unstructured data store after the raw machine data has been stored in the unstructured data store, to impose a structure on the raw machine data.
17. A system comprising:
a processor; and
a memory storing instructions, execution of which by the processor cause the system to perform operations comprising:
obtaining, by a query converter in the system, a structured query generated by an application in a structured query language, the structured query representing a requested search of raw machine data, the raw machine data being stored in an unstructured data store as unstructured raw machine data that has been segmented and timestamped;
identifying, by the query converter, a first set of fields in the unstructured data to obtain field identification data from the unstructured data, the unstructured data including records, each of the fields in the first set of fields corresponding to a portion of information extracted from a portion of at least one of the records;
generating, by the query converter, an unstructured query in a second query language associated with the unstructured data store, based on the structured query, wherein generating the unstructured query in the second query language includes using the identified first set of fields to generate the unstructured query;
causing execution of the unstructured query against the unstructured data stored in the unstructured data store;
receiving a result of execution of the unstructured query against the unstructured data stored in the unstructured data store; and
causing an indication of the result to be provided to the application, wherein the indication of the result is provided to the application as a direct response to the structured query without requiring any additional query from the application.
18. The system of claim 17 , wherein the unstructured raw machine data has been segmented into a plurality of events prior to said receiving the structured query;
said operations further comprising:
identifying a value of a field in an event stored in the unstructured data store, based on an extraction rule that specifies where to find a subportion of text within an event.
19. A system of claim 17 , wherein the unstructured raw machine data has been segmented into a plurality of events prior to said obtaining the structured query;
said operations further comprising:
identifying a field in an event stored in the unstructured data store;
wherein generating the unstructured query in the second query language associated with the unstructured data store is based on an identification of the field.
20. The system of claim 17 , wherein the unstructured query causes one or more values for one or more fields included in the unstructured query to be extracted as a function of a format of the unstructured data.