Automatic partitioning

1. A system, comprising:

one or more processors configured to:

receive a set of previously run queries for selecting logs from log data, each log comprising data collected during operation of one or more computer systems;

determine a set of query fragments contained in the set of previously run queries, wherein at least one query fragment in the set of query fragments includes less than all words in one of the previously run queries;

access a set of logs in the log data;

for each log in the set of logs:

determine which query fragments, in the set of query fragments, are contained in the log; and

generate a bitset for the log based on the query fragments contained in the log, wherein each bit in the bitset corresponds to one query fragment in the set of query fragments, wherein each bit in the bitset indicates whether the log contains the query fragment associated with the bit;

for each generated bitset, determine a number of logs in the set of logs that match the bitset;

for each generated bitset, based on the determined number of logs for the bitset, generate a partition in the log data for the logs having the bitset, wherein logs having the bitset are placed in the partition; and

index the logs in the log data belonging to the partition; and

a memory coupled to the one or more processors and configured to provide the one or more processors with instructions.

2. The system of claim 1 , wherein the set of query fragments comprises fragments of queries determined to be common among the set of previously run queries.

3. The system of claim 2 , wherein a query fragment is determined to be common based at least in part on an analysis of a frequency of occurrence of the query fragment in the set of previously run queries.

4. The system of claim 1 , wherein the one or more processors are further configured to perform partition rebuilding.

5. The system of claim 4 , wherein the partition rebuilding is performed periodically.

6. The system of claim 4 , wherein the partition rebuilding is performed based at least in part on an evaluation of a second set of previously run queries.

7. The system of claim 1 , wherein generating the partition includes evaluating a plurality of candidate partition sets.

8. The system of claim 7 , wherein evaluating the plurality of candidate partition sets includes estimating a cost associated with building a given candidate partition set.

9. The system of claim 7 , wherein evaluating the plurality of candidate partition sets includes analyzing a savings associated with building a given candidate partition set.

10. The system of claim 1 , wherein indexing the logs in the log data includes, for each log:

determining that the log has the bitset associated with the partition; and

adding the log to the partition in response to the determination.

11. The system of claim 1 , wherein the one or more processors are further configured to receive a query for log data.

12. The system of claim 11 , wherein the one or more processors are further configured to rewrite the query based at least in part on the generated partition.

13. The system of claim 12 , wherein the one or more processors are further configured to use the rewritten query to search the generated partition.

14. The system of claim 1 wherein the one or more processors are further configured to:

determine a partition recommendation based at least in part on the set of query fragments;

provide the partition recommendation to a user for review; and

receive user input indicating approval of the partition recommendation, wherein the partition is generated based on the received user approval.

15. A method, comprising:

receiving a set of previously run queries for selecting logs from log data, each log comprising data collected during operation of one or more computer systems;

determining a set of query fragments contained in the set of previously run queries, wherein at least one query fragment in the set of query fragments includes less than all words in one of the previously run queries;

accessing a set of logs in the log data;

for each log in the set of logs:

determining which query fragments, in the set of query fragments, are contained in the log; and

generating a bitset for the log based on the query fragments contained in the log, wherein each bit in the bitset corresponds to one query fragment in the set of query fragments, wherein each bit in the bitset indicates whether the log contains the query fragment associated with the bit;

for each generated bitset, determine a number of logs in the set of logs that match the bitset;

for each generated bitset, based on the determined number of logs for each bitset, generating, using one or more processors, a partition in the log data for the logs having the bitset, wherein logs having the bitset are placed in the partition; and

indexing the logs in the log data belonging to the partition.

16. A computer program product embodied in a non-transitory computer readable storage medium and comprising computer instructions for:

receiving a set of previously run queries for selecting logs from log data, each log comprising data collected during operation of one or more computer systems;

determining a set of query fragments contained in the set of previously run queries, wherein at least one query fragment in the set of query fragments includes less than all words in one of the previously run queries;

accessing a set of logs in the log data;

for each log in the set of logs:

determining which query fragments, in the set of query fragments, are contained in the log; and

generating a bitset for the log based on the query fragments contained in the log, wherein each bit in the bitset corresponds to one query fragment in the set of query fragments, wherein each bit in the bitset indicates whether the log contains the query fragment associated with the bit;

for each generated bitset, determine a number of logs in the set of logs that match the bitset;

for ach generated bitset, based on the determined number of logs for each bitset, generating a partition in the log data for the logs having the bitset, wherein logs having the bitset are placed in the partition; and

indexing the logs in the log data belonging to the partition.