Automatic generation of regular expression based on log line data
In one aspect, a regular expression is automatically generated based on user input for fields that are desired to be extracted from log lines. The input may be received by user through an interface provided by a machine such as a controller. The input may identify one or more fields within a log line that should be extracted. Multiple instances of potential regular expression portions may be generated based on the user input, and different portions are combined together to determine if they achieve the desired extraction. Once a complete regular expression is generated based on user input, a user may provide additional input to identify examples or counterexamples of log line fields that satisfy or don't satisfy the user's intended extraction.
1. A method for automatically generating a regular expression from log line data, comprising:
identifying, by an application on a server, a portion of a log line in the log line data;
automatically generating, by the application on the server, multiple instances of multiple portions of a regular expression based on the identified portion, the multiple instances of multiple portions of the regular expression including:
at least one regular expression generated that represents a field within the portion of the log line, and
at least one regular expression that represents a gap within the portion of the log line, where the at least one regular expression that represents the gap is generated by identifying starting patterns and ending patterns of the gap, a particular pattern for a string immediately after the gap, or a count of delimiter occurrences defining the gap;
automatically generating, by the application on the server, a complete regular expression from one or more of the generated multiple instances; and
retrieving, by the application on the server, subsequent log lines based on the automatically generated complete regular expression.
2. The method of claim 1 , wherein identifying includes receiving user input indicative of the portion of a log line.
3. The method of claim 2 , wherein the received user input identifies a field in the indicated portion of the log line.
4. The method of claim 1 , wherein generating multiple instances includes:
generating at least one of the multiple instances of the multiple portions of the regular expressions for each of a plurality of gaps in the log line based on the identified portion; and
generating at least one of the multiple instances of the multiple portions of the regular expressions for each of a plurality of fields in the log line based on the identified portion.
5. The method of claim 4 , wherein automatically generating a regular expression includes generating a regular expression based on multiple instances of the regular expression portion for a single gap and multiple instances of the regular expression portion for a single field.
6. The method of claim 5 , further comprising receiving user input to modify the automatically generated regular expression based on receipt of a subsequent user input.
7. The method of claim 1 , further comprising receiving user input to modify the automatically generated regular expression based on receipt of a subsequent user input.
8. A non-transitory computer readable storage medium having embodied thereon a program, the program being executable by a processor to perform a method for automatically generating a regular expression from log line data, the method comprising:
identifying, by an application on a server, a portion of a log line in the log line data;
automatically generating, by the application on the server, multiple instances of multiple portions of a regular expression based on the identified portion, the multiple instances of multiple portions of the regular expression including:
at least one regular expression generated that represents a field within the portion of the log line, and
at least one regular expression that represents a gap within the portion of the log line, where the at least one regular expression that represents the gap is generated by identifying starting patterns and ending patterns of the gap, a particular pattern for a string immediately after the gap, or a count of delimiter occurrences defining the gap;
automatically generating, by the application on the server, a complete regular expression from one or more of the generated multiple instances; and
retrieving, by the application on the server, subsequent log lines based on the automatically generated complete regular expression.
9. The non-transitory computer readable storage medium of claim 8 , wherein identifying includes receiving user input indicative of the portion of a log line.
10. The non-transitory computer readable storage medium of claim 9 , wherein the received user input identifies a field in the indicated portion of the log line.
11. The non-transitory computer readable storage medium of claim 8 , wherein generating multiple instances includes:
generating at least one of the multiple instances of the multiple portions of the regular expressions for each of a plurality of gaps in the log line based on the identified portion; and
generating at least one of the multiple instances of the multiple portions of the regular expressions for each of a plurality of fields in the log line based on the identified portion.
12. The non-transitory computer readable storage medium of claim 11 , wherein automatically generating a regular expression includes generating a regular expression based on multiple instances of the regular expression portion for a single gap and multiple instances of the regular expression portion for a single field.
13. The non-transitory computer readable storage medium of claim 12 , further comprising receiving user input to modify the automatically generated regular expression based on receipt of a subsequent user input.
14. The non-transitory computer readable storage medium of claim 8 , further comprising receiving user input to modify the automatically generated regular expression based on receipt of a subsequent user input.
15. A system for automatically generating a regular expression from log line data, comprising:
a server including a memory and a processor; and
one or more modules stored in the memory and executed by the processor to:
identify a portion of a log line in the log line data,
automatically generate multiple instances of multiple portions of a regular expression based on the identified portion, the multiple instances of multiple portions of the regular expression including:
at least one regular expression generated that represents a field within the portion of the log line, and
at least one regular expression that represents a gap within the portion of the log line, where the at least one regular expression that represents the gap is generated by identifying starting patterns and ending patterns of the gap, a particular pattern for a string immediately after the gap, or a count of delimiter occurrences defining the gap,
automatically generate a complete regular expression from one or more of the generated multiple instances, and
retrieve subsequent log lines based on the automatically generated complete regular expression.
16. The system of claim 15 , wherein identifying includes receiving user input indicative of the portion of a log line.
17. The system of claim 16 , wherein the received user input identifies a field in the indicated portion of the log line.
18. The system of claim 15 , wherein the one or more modules are further executable to generate at least one of the multiple instances of the multiple portions of the regular expressions for each of a plurality of gaps in the log line based on the identified portion, and generate at least one of the multiple instances of the multiple portions of the regular expressions for each of a plurality of fields in the log line based on the identified portion.
19. The system of claim 18 , wherein generating a regular expression includes generating a regular expression based on multiple instances of the regular expression portion for a single gap and multiple instances of the regular expression portion for a single field.
20. The system of claim 19 , wherein the one or more modules are further executable to receive user input to modify the automatically generated regular expression based on receipt of a subsequent user input.
21. The system of claim 15 , wherein the one or more modules are further executable to receive user input to modify the automatically generated regular expression based on receipt of a subsequent user input.