Using anchors to generate extraction rules
The technology disclosed relates to formulating and refining field extraction rules that are used at query time on raw data with a late-binding schema. The field extraction rules identify portions of the raw data, as well as their data types and hierarchical relationships. These extraction rules are executed against very large data sets not organized into relational structures that have not been processed by standard extraction or transformation methods. By using sample events, a focus on primary and secondary example events help formulate either a single extraction rule spanning multiple data formats, or multiple rules directed to distinct formats. Selection tools mark up the example events to indicate positive examples for the extraction rules, and to identify negative examples to avoid mistaken value selection. The extraction rules can be saved for query-time use, and can be incorporated into a data model for sets and subsets of event data.
1. A computer-implemented method comprising:
causing display, via a graphical user interface, of an example event having a set of field values corresponding with fields, wherein a first field value, corresponding with a first field, is visually indicated as corresponding with a field extraction rule that extracts one or more values for one or more fields;
based on an indication that the field extraction rule fails to extract a second field value associated with a second field in the example event, receiving an input indicating to anchor the second field to the first field, wherein the second field and the first field are adjacent to one another; and
generating a new field extraction rule configured to extract the first field value of the first field and the second field value of the second field of the example event based on the second field being anchored to the first field.
2. The method of claim 1 , wherein the example event includes a portion of raw data and a timestamp.
3. The method of claim 1 further comprising causing display, via the graphical user interface, of the field extraction rule.
4. The method of claim 1 further comprising selecting the example event for which to apply the field extraction rule.
5. The method of claim 1 further comprising applying the field extraction rule to the example event.
6. The method of claim 1 further comprising applying the new field extraction rule to the example event.
7. The method of claim 1 further comprising:
applying the new field extraction rule to the example event; and
causing display, via the graphical user interface, of the example event including the first field value and the second field value being visually indicated.
8. The method of claim 1 , wherein the new field extraction rule extracts the second field value by selecting one or more characters after the first field value.
9. The method of claim 1 , wherein the new field extraction rule extracts the second field value by selecting characters after the first field value based on an extraction rule.
10. A system comprising:
one or more data processors; and
one or more computer-readable storage media containing instructions which when executed on the one or more data processors, cause the one or more data processors to perform operations including:
causing display, via a graphical user interface, of an example event having a set of field values corresponding with fields, wherein a first field value, corresponding with a first field, is visually indicated as corresponding with a field extraction rule that extracts one or more values for one or more fields;
based on an indication that the field extraction rule fails to extract a second field value associated with a second field in the example event, receiving an input indicating to anchor the second field to the first field, wherein the second field and the first field are adjacent to one another; and
generating a new field extraction rule configured to extract the first field value of the first field and the second field value of the second field of the example event based on the second field being anchored to the first field.
11. The system of claim 10 , wherein the example event includes a portion of raw data and a time stamp.
12. The system of claim 10 further comprising causing display, via the graphical user interface, of the field extraction rule.
13. The system of claim 10 further comprising selecting the example event for which to apply the field extraction rule.
14. The system of claim 10 further comprising applying the field extraction rule to the example event.
15. One or more computer-storage media storing computer-executable instructions that, when executed by a computing device, perform a method comprising:
causing display, via a graphical user interface, of an example event having a set of field values corresponding with fields, wherein a first field value, corresponding with a first field, is visually indicated as corresponding with a field extraction rule that extracts one or more values for one or more fields;
based on an indication that the field extraction rule fails to extract a second field value associated with a second field in the example event, receiving an input indicating to anchor the second field to the first field, wherein the second field and the first field are adjacent to one another; and
generating a new field extraction rule configured to extract the first field value of the first field and the second field value of the second field of the example event based on the second field being anchored to the first field.
16. The one or more computer-storage media of claim 15 further comprising applying the new field extraction rule to the example event.
17. The one or more computer-storage media of claim 15 further comprising:
applying the new field extraction rule to the example event; and
causing display, via the graphical user interface, of the example event including the first field value and the second field value being visually indicated.
18. The one or more computer-storage media of claim 15 , wherein the new field extraction rule extracts the second field value by selecting one or more characters after the first field value.
19. The one or more computer-storage media of claim 15 , wherein the new field extraction rule extracts the second field value by selecting characters after the first field value based on an extraction rule.
20. The one or more computer-storage media of claim 15 wherein the example event includes a portion of raw data and a timestamp.