Query summary generation using row-column data storage

1. A method of generating a query summary from data stored in a column store and a row store in a data storage system, the method comprising:

receiving a query;

determining query summary fields comprised of a subset of fields from a schema used to store data in the row store;

determining if all the query summary fields are stored in the column store;

if all the query summary fields are stored in the column store, searching the column store for query results in the query summary fields;

if one of the query summary fields is not stored in the column store, searching the row store instead of the column store for the query results in the query summary fields;

generating, by a processor, a query summary from the query results in the query summary fields;

receiving a drill down request pertaining to the query summary, the drill down request identifying a query summary field;

determining that the query summary field is not a field of the schema used to store data in the row store;

in response to determining that the query summary field is not a field of the schema, creating a discovered field and adding the discovered field to the row store; and

searching the row store using tokens associated with the discovered field to retrieve events matching the query for the query summary field, to satisfy the drill down request.

2. The method of claim 1 , wherein generating a query summary from the query results in the query summary fields comprises:

determining counts of field values in each of the query summary fields; and

providing the counts in the query summary.

3. The method of claim 1 , wherein generating a query summary from the query results in the query summary fields comprises:

bucketizing, by the processor, field values for a query summary field into buckets;

determining aggregated counts of the field values in the buckets; and

providing the aggregated counts in the query summary.

4. The method of claim 1 , wherein bucketizing field values for a query summary field into buckets comprises determining buckets by increasing a log scale if uniform buckets are scored.

5. The method of claim 1 , wherein generating a query summary from the query results in the query summary fields comprises:

determining an approximate count of field values in a query summary field, wherein determining an approximate count comprises:

determining a predetermined number of field values for the query summary field;

receiving a field value not in the predetermined number of field values;

determining using a bloom filter for the query summary field, whether the received field value is a field value that was received previously;

if the field value is determined to have been received previously, identifying a field value from the predetermined number of field values having a lowest count, discarding the field value having the lowest count, including the received field value in the predetermined number of field values, and assigning a count to the received value equal to the lowest count plus one; and

if the field value is determined to not have been received previously, identifying a field value from the predetermined number of field values having a lowest count, discarding the field value having a lowest count, including the received field value in the predetermined number of field values, and assigning a count to the received value equal to one.

6. The method of claim 1 , wherein determining the query summary fields comprises:

prioritizing the query summary fields based on skew and frequency of hits in the query summary fields; and

generating the query summary according to the priorities of the query summary fields.

7. The method of claim 1 , comprising:

identifying specific field values in the query results that are the query results more than a predetermined number of times;

determining correlations between the specific field values;

generating tuples of the specific field values based on the correlations; and

generating a tuple display for the tuples, the tuple display comprising a list of the tuples, a tree-based display for each tuple, or a table display of tuples in each tree.

8. The method of claim 1 , comprising:

generating episodic global summaries;

using the episodic global summaries to optimize the query if the query is time-based.

9. A non-transitory computer readable medium including machine readable instructions that when executed by a processor perform a method of generating a query summary from data stored in a column store and a row store in a data storage system, the instructions comprising instructions to:

receive a query;

determine query summary fields comprised of a subset of fields from a schema used to store data in the row store;

determine if all the query summary fields are stored in the column store;

if all the query summary fields are stored in the column store, search the column store for query results in the query summary fields;

if one of the query summary fields is not stored in the column store, search the row store instead of the column store for the query results in the query summary fields;

generate a query summary from the query results in the query summary fields;

receive a drill down request pertaining to the query summary, the drill down request identifying a query summary field;

determine that the query summary field is not a field of the schema used to store data in the row store;

in response to determining that the query summary field is not a field of the schema, create a discovered field and adding the discovered field to the row store; and

search the row store using tokens associated with the discovered field to retrieve events matching the query for the query summary field, to satisfy the drill down request.

10. The non-transitory computer readable medium of claim 9 , wherein instructions to generate a query summary from the query results in the query summary fields comprise instructions to:

determine counts of field values in each of the query summary fields; and

provide the counts in the query summary.

11. The non-transitory computer readable medium of claim 9 , wherein instructions to generate a query summary from the query results in the query summary fields comprise instructions to:

bucketize field values for a query summary field into buckets;

determine aggregated counts of the field values in the buckets; and

provide the aggregated counts in the query summary.

12. The non-transitory computer readable medium of claim 9 , wherein instructions to bucketize field values for a query summary field into buckets comprise instructions to determine buckets by increasing a log scale if uniform buckets are scored.

13. The non-transitory computer readable medium of claim 9 , wherein instructions to generate a query summary from the query results in the query summary fields comprise instructions to:

determine an approximate count of field values in a query summary field, wherein determining an approximate count comprises:

determining a predetermined number of field values for the query summary field;

receiving a field value not in the predetermined number of field values;

determining using a bloom filter for the query summary field, whether the received field value is a field value that was received previously;

if the field value is determined to have been received previously, identifying a field value from the predetermined number of field values having a lowest count, discarding the field value having the lowest count, including the received field value in the predetermined number of field values, and assigning a count to the received value equal to the lowest count plus one; and

if the field value is determined to not have been received previously, identifying a field value from the predetermined number of field values having a lowest count, discarding the field value having a lowest count, including the received field value in the predetermined number of field values, and assigning a count to the received value equal to one.

14. A data storage system comprising:

a processor to

receive a query;

determine query summary fields comprised of a subset of fields from a schema used to store data in the row store;

determine if all the query summary fields are stored in the column store;

if all the query summary fields are stored in the column store, search the column store for query results in the query summary fields;

if one of the query summary fields is not stored in the column store, search the row store instead of the column store for the query results in the query summary fields;

generate a query summary from the query results in the query summary fields;

receive a drill down request pertaining to the query summary, the drill down request identifying a query summary field;

determine that the query summary field is not a field of the schema used to store data in the row store;

in response to determining that the query summary field is not a field of the schema, create a discovered field and adding the discovered field to the row store; and

search the row store using tokens associated with the discovered field to retrieve events matching the query for the query summary field, to satisfy the drill down request; and

a memory to store the query summary.