Iterative building of search expressions to match specified string values
Methods and systems for generating a search expression. The system begins with an empty search expression, and iteratively expands the search expression until some terminating condition is reached.
1. A method comprising:
performing, by one or more computing devices that implements a search expression generator:
determining an initial expression to begin generation of a search expression, wherein the search expression is to be generated based on desired instances of a field in a training set of text strings;
performing an iterative process, wherein individual iterations of the iterative process comprises:
expanding the search expression with at least one of a positive character set, a negative character set, or a quantifier based on samples in the training set,
evaluating the expanded search expression to determine how well the search expression matches the desired instances of the field in the training set, and
rejecting the expansion if the expanded search expression matches the desired instances of the field less accurately than the search expression prior to the expansion; and
terminating the iterative process when a terminating condition is reached.
2. The method of claim 1 , wherein an individual iteration of the iterative process comprises:
generating multiple expanded search expressions from the search expression; and
selecting one or more of the multiple expanded search expressions for one or more later iterations of the iterative process.
3. The method of claim 2 , wherein the selecting of the one or more expanded search expression for the one or more later iteration is based on one or more of:
a length of the expanded search expression,
a complexity of the expanded search expression,
a generality of the expanded search expression, and
a computational effectiveness of the search expression.
4. The method of claim 1 , wherein the iterative process outputs a list of search expressions generated by at least some of the iterations.
5. The method of claim 4 , wherein
the list of search expressions is outputted via a user interface, and
the initial expression is received via the user interface and selected from another list of search expressions generated previously by the search expression generator.
6. The method of claim 1 , wherein the initial expression is an empty expression.
7. The method of claim 1 , wherein
the search expression generator implements a user interface, and
the desired instances of the field in the training set are received via the user interface.
8. The method of claim 1 , wherein the terminating condition of the iterative process is based on a maximum length of the search expression or a maximum number of iterations of the iterative process.
9. The method of claim 1 , further comprising search expression generator:
storing a set of generated search expressions that matches the desired instances of the field in the training set, wherein the stored set of search expressions is used to detect additional text strings that contain the desired instances of the field.
10. The method of claim 9 , wherein the training set is a log file of unstructured log lines.
11. A system comprising:
one or more processors; and
a memory coupled to the one or more processors, wherein the memory stores program instructions executable by the one or more processors to implement a search expression generator that is configured to:
determine an initial expression to begin generation of a search expression, wherein the search expression is to be generated based on desired instances of a field in a training set of text strings;
perform an iterative process, wherein individual iterations of the iterative process comprise:
expanding the search expression with at least one of a positive character set, a negative character set, or a quantifier based on samples in the training set,
evaluating the expanded search expression to determine how well the search expression matches the desired instances of the field in the training set, and
rejecting the expansion if the expanded search expression matches the desired instances of the field less accurately than the search expression prior to the expansion; and
terminate the iterative process when a terminating condition is reached.
12. The system of claim 11 , wherein an individual iteration of the iterative process comprises:
generate multiple expanded search expressions from the search expression; and
select one or more of the multiple expanded search expressions for one or more later iterations of the iterative process.
13. The system of claim 12 , wherein the selection of the one or more expanded search expression for the one or more later iteration is based on user configuration input received by the search expression generator.
14. The system of claim 12 , wherein the selection of the one or more expanded search expression for the one or more later iteration is performed based on an A* search algorithm.
15. The system of claim 11 , wherein the iterative process outputs a list of search expression s generated by at least some of the iterations.
16. The system of claim 15 , wherein
the list of search expressions is outputted via a user interface of the search expression generator, and
the initial expression is received via the user interface and selected from another list of search expressions generated previously by the search expression generator.
17. The system of claim 11 , wherein
the initial expression is a regular expression, and
an expansion of the search expression comprises modifying the regular expression to one or more additional characters.
18. The system of claim 11 , wherein the training set includes text strings with no instances of the field to be matched by the search expression to act as counter examples.
19. The system of claim 11 , wherein the terminating condition of the iterative process is based on a maximum length of the search expression or a maximum number of iterations of the iterative process.
20. The system of claim 11 , wherein the search expression generator is configured to:
store a set of generated search expressions that matches the desired instances of the field in the training set, wherein the stored set of search expressions is used to detect log lines in a log file that contain the desired instances of the field.