Exploratory data analysis system for automated generation of search queries using machine learning techniques to identify certain log fields and correlation thereof
Implementations of this disclosure provide a search assistant engine that integrates with a data intake and query system and provides an intuitive user interface to assist a user in searching and evaluating indexed event data. Additionally, the search assistant engine provides logic to intelligently provide data to the user through the user interface such as determining fields of events likely to be of interest based on determining a mutual information score for each field and determining groups of related fields based on determining a mutual information score for each field grouping. Some implementations utilize machine learning techniques in certain analyses such as when clustering events and determining an event templates for each cluster. Additionally, the search assistant engine may import terms or characters from user interaction into predetermined search query templates to generate tailored search query for the user.
1. A computerized method comprising:
obtaining event data including a plurality of events;
determining a set of fields present in events comprising the plurality of events;
determining a mutual information score for a plurality of fields included within the set of fields, wherein the mutual information score for the plurality of fields identifies a level of diversity between values included in the plurality of fields;
causing display of a user interface that illustrates at least a portion of the mutual information score for the plurality of fields;
receiving user input through the user interface indicating selection of a first field of the plurality of fields of the set of fields; and
importing the selected first field into a predetermined search query template thereby generating a tailored search query.
2. The computerized method of claim 1 , further comprising:
executing the tailored search query thereby generating search query results.
3. The computerized method of claim 1 , further comprising:
parsing the plurality of events and maintaining a count for each value of the selected first field;
designating a subset of the values of the selected first field as rare values; and
causing display of at least a first event including a first rare value.
4. The computerized method of claim 1 , wherein the determining of the mutual information score for the plurality of fields comprises
determining the mutual information score for a plurality of field groupings, wherein each field grouping of the plurality of field groupings includes at least two fields; and
causing display of at least a first field grouping of the plurality of field groupings and a corresponding mutual information score.
5. The computerized method of claim 1 , further comprising:
obtaining a count of each value-value pairing for two fields of the plurality of fields; and
causing display of at least a first value-value pairing for the first field and a second value and a corresponding count.
6. The computerized method of claim 1 , wherein the event data is retrieved from an index specified by user input received by the user interface.
7. The computerized method of claim 1 , wherein the event data is filtered to include events derived from a source type specified by user input received by the user interface.
8. A computing device, comprising:
a processor; and
a non-transitory computer-readable medium having stored thereon instructions that, when executed by the processor, cause the processor to perform operations including:
obtaining event data including a plurality of events;
determining a set of fields present in events comprising the plurality of events;
determining a mutual information score for a plurality of fields included within the set of fields, wherein the mutual information score for the plurality of fields identifies a level of diversity between values included in the plurality of fields;
causing display of a user interface that illustrates at least a portion of the mutual information score for the plurality of fields;
receiving user input though the user interface indicating selection of a first field of the plurality of fields of the set of fields; and
importing the selected first field into a predetermined search query template thereby generating a tailored search query.
9. The computing device of claim 8 , wherein the operations further include:
executing the tailored search query thereby generating search query results.
10. The computing device of claim 8 , wherein the operations further include:
parsing the plurality of events and maintaining a count for each value of the selected first field;
designating a subset of the values of the selected first field as rare values; and
causing display of at least a first event including a first rare value.
11. The computing device of claim 8 , wherein the determining of the mutual information score for the plurality of fields further comprises:
determining the mutual information score for a plurality of field groupings, wherein each field grouping of the plurality of field groupings includes at least two fields; and
causing display of at least a first field grouping of the plurality of field groupings and a corresponding mutual information score.
12. The computing device of claim 8 , wherein the operations further include:
obtaining a count of each value-value pairing for two fields of the plurality of fields; and
causing display of at least a first value-value pairing for the first field and a second value and a corresponding count.
13. The computing device of claim 8 , wherein the event data is retrieved from an index specified by user input received by the user interface.
14. The computing device of claim 8 , wherein the event data is filtered to include events derived from a source type specified by user input received by the user interface.
15. A non-transitory computer-readable medium having stored thereon instructions that, when executed by one or more processors, cause the one or more processor to perform operations comprising:
obtaining event data including a plurality of events;
determining a set of fields present in events comprising the plurality of events;
determining a mutual information score for a plurality of fields included within the set of fields, wherein the mutual information score for the plurality of fields identifies a level of diversity between values included in the plurality of fields;
causing display of a user interface that illustrates at least a portion of the mutual information score for the plurality of fields;
receiving user input though the user interface indicating selection of a first field of the plurality of fields of the set of fields; and
importing the selected first field into a predetermined search query template thereby generating a tailored search query.
16. The non-transitory computer-readable medium of claim 15 , wherein the operations further include:
executing the tailored search query thereby generating search query results.
17. The non-transitory computer-readable medium of claim 15 , wherein the operations further include:
parsing the plurality of events and maintaining a count for each value of the selected first field;
designating a subset of the values of the selected first field as rare values; and
causing display of at least a first event including a first rare value.
18. The non-transitory computer-readable medium of claim 15 , wherein the determining of the mutual information score for the plurality of fields further comprises:
determining the mutual information score for a plurality of field groupings, wherein each field grouping of the plurality of field groupings includes at least two fields; and
causing display of at least a first field grouping of the plurality of field groupings and a corresponding mutual information score.
19. The non-transitory computer-readable medium of claim 15 , wherein the operations further include:
obtaining a count of each value-value pairing for two fields of the plurality of fields; and
causing display of at least a first value-value pairing for the first field and a second value and a corresponding count.
20. The non-transitory computer-readable medium of claim 15 , wherein the event data is retrieved from an index specified by user input received by the user interface, and wherein the event data is filtered to include events derived from a source type specified by user input received by the user interface.