Compression of machine-generated data

1. A method, comprising:

receiving a pre-shared compression dictionary, wherein the pre-shared compression dictionary was generated based on an analysis of sample data for use in compression of other data;

receiving a compressed version of a batch of machine-generated data, wherein the batch of machine-generated data has been compressed together at least in part using the pre-shared compression dictionary and a batch-specific compression dictionary, wherein the pre-shared compression dictionary is shared across a plurality different batches and the batch-specific compression dictionary is generated based on content of an intermediate version of a single batch among the plurality different batches that share the pre-shared compression dictionary, and the compressed version of the batch of machine-generated data has been compressed together using at least two different compression dictionaries including the pre-shared compression dictionary and the batch-specific compression dictionary;

uncompressing the received compressed version of the batch using the batch-specific compression dictionary different from the pre-shared compression dictionary to determine the intermediate version; and

uncompressing the intermediate version using the pre-shared compression dictionary to determine an uncompressed version of the batch of machine-generated data.

2. The method of claim 1 , wherein the machine-generated data includes computer log entries.

3. The method of claim 1 , wherein the sample data analyzed to generate the pre-shared compression dictionary includes contents of previously sent batches of a same source of the machine-generated data.

4. The method of claim 1 , wherein the sample data analyzed to generate the pre-shared compression dictionary includes contents of the batch.

5. The method of claim 1 , wherein the sample data analyzed to generate the pre-shared compression dictionary includes contents of a plurality different log files.

6. The method of claim 1 , wherein the pre-shared compression dictionary is received separately from the compressed version of the batch.

7. The method of claim 1 , wherein the analysis of the sample data includes performing identifications of portions of the sample data that are repeated in the sample data.

8. The method of claim 7 , wherein the analysis of the sample data includes ranking the identified repeated portions on character length and a number of repeats in the sample data.

9. The method of claim 1 , wherein the analysis of the sample data included a determination of suffix trees and/or suffix arrays for the sample data.

10. The method of claim 1 , further comprising storing the uncompressed version of the batch of machine-generated data in an archival storage.

11. The method of claim 1 , further comprising analyzing the uncompressed version of the batch of machine-generated data.

12. A system, comprising:

one or more processors configured to:

receive a pre-shared compression dictionary, wherein the pre-shared compression dictionary was generated based on an analysis of sample data for use in compression of other data;

receive a compressed version of a batch of machine-generated data, wherein the batch of machine-generated data has been compressed together at least in part using the pre-shared compression dictionary and a batch-specific compression dictionary, wherein the pre-shared compression dictionary is shared across a plurality different batches and the batch-specific compression dictionary is generated based on content of an intermediate version of a single batch among the plurality different batches that share the pre-shared compression dictionary, and the compressed version of the batch of machine-generated data has been compressed together using at least two different compression dictionaries including the pre-shared compression dictionary and the batch-specific compression dictionary

uncompress the received compressed version of the batch using the batch-specific compression dictionary different from the pre-shared compression dictionary to determine the intermediate version; and

uncompress the intermediate version using the pre-shared compression dictionary to determine an uncompressed version of the batch of machine-generated data; and

a memory coupled to the one or more processors and configured to provide the one or more processors with instructions.

13. A method, comprising:

determining to send a batch of machine-generated data;

determining whether a valid pre-shared compression dictionary has been provided to a destination for the machine-generated data;

determining that the valid pre-shared compression dictionary has been previously provided to the destination for the machine-generated data;

compressing together the batch of machine-generated data using the valid pre-shared compression dictionary to generate an intermediate version, wherein the valid pre-shared compression dictionary was generated based on an analysis of sample data that did not include contents of the batch of machine-generated data;

compressing the intermediate version using a batch-specific compression dictionary different from the pre-shared compression dictionary to generate an output compressed version, wherein the pre-shared compression dictionary is shared across a plurality different batches and the batch-specific compression dictionary generated based on content of the intermediate version of a single batch among the plurality different batches that share the pre-shared compression dictionary, and the output compressed version has been compressed using at least two different compression dictionaries including the pre-shared compression dictionary and the batch-specific compression dictionary; and

sending to the destination the output compressed version.

14. The method of claim 13 , wherein the machine-generated data includes computer log entries.

15. The method of claim 13 , wherein the sample data analyzed to generate the valid pre-shared compression dictionary includes contents of previously sent batches of a same source of the machine-generated data.

16. The method of claim 13 , wherein the sample data analyzed to generate the valid pre-shared compression dictionary includes contents of a plurality different log files.

17. The method of claim 13 , wherein the analysis of the sample data includes performing identifications of portions of the sample data that are repeated in the sample data.

18. The method of claim 17 , wherein the analysis of the sample data includes ranking the identified repeated portions on character length and a number of repeats in the sample data.

19. The method of claim 13 , wherein the analysis of the sample data included a determination of suffix trees and/or suffix arrays for the sample data.

20. The method of claim 13 , wherein the output compressed version is sent to a remote system for archival or analysis.