System and method for investigating large amounts of data
A data analysis system is proposed for providing fine-grained low latency access to high volume input data from possibly multiple heterogeneous input data sources. The input data is parsed, optionally transformed, indexed, and stored in a horizontally-scalable key-value data repository where it may be accessed using low latency searches. The input data may be compressed into blocks before being stored to minimize storage requirements. The results of searches present input data in its original form. The input data may include access logs, call data records (CDRs), e-mail messages, etc. The system allows a data analyst to efficiently identify information of interest in a very large dynamic data set up to multiple petabytes in size. Once information of interest has been identified, that subset of the large data set can be imported into a dedicated or specialized data analysis system for an additional in-depth investigation and contextual analysis.
1. A computer-implemented method comprising:
receiving a stream of input data;
parsing the input data to identify boundaries of logical data entities in the stream of input data;
generating a block item comprising a key-value family identifier, a data block identifier, and a data block based on a logical data entity of the logical data entities, comprising:
compressing the data block; and
storing the key-value family identifier in association with a key-value pair comprising the data block identifier as a key of the key-value pair and the compressed data block as a value of the key-value pair;
creating and storing a parse item comprising the data block identifier and one or more parse tokens for indexing block items.
2. The computer-implemented method of claim 1 , further comprising:
receiving a search criterion;
using the parse item, determining that the one or more parse tokens match the search criterion;
using the data block identifier as a key to the key-value pair, identifying the compressed data block;
uncompressing the data block;
using the search criterion to identify one or more portions of the uncompressed data block; and
returning the one or more portions of the uncompressed data block as search results.
3. The computer-implemented method of claim 1 , further comprising storing a plurality of key-value pairs for a plurality of compressed data blocks, wherein each key-value pair of the plurality of key-value pairs is unique at least amongst all key-value pairs of the plurality of key-value pairs.
4. The computer-implemented method of claim 3 , wherein the plurality of key-value pairs comprises at least one million unique keys.
5. The computer-implemented method of claim 1 , wherein creating the parse item comprises extracting the one or more parse tokens from one or more of the logical data entities.
6. The computer-implemented method of claim 1 , wherein the parse item further comprises snippet identifying information.
7. The computer-implemented method of claim 6 , wherein the snippet identifying information is a byte offset into an uncompressed block and a byte length.
8. The computer-implemented method of claim 1 , wherein the parse tokens identify byte-sequential portions of the data block.
9. A system comprising:
one or more processors;
a memory storing instructions which, when executed by the one or more processors, causes performing:
receiving a stream of input data;
parsing the input data to identify boundaries of logical data entities in the stream of input data;
generating a block item comprising a key-value family identifier, a data block identifier, and a data block based on a logical data entity of the logical data entities, comprising:
compressing the data block; and
storing the key-value family identifier in association with a key-value pair comprising the data block identifier as a key of the key-value pair and the compressed data block as a value of the key-value pair;
creating and storing a parse item comprising the data block identifier and one or more parse tokens for indexing block items.
10. The system of claim 9 , wherein the instructions, when executed by the one or more processors, further cause performing:
receiving a search criterion;
using the parse item, determining that the one or more parse tokens match the search criterion;
using the data block identifier as a key to the key-value pair, identifying the compressed data block;
uncompressing the data block;
using the search criterion to identify one or more portions of the uncompressed data block; and
returning the one or more portions of the uncompressed data block as search results.
11. The system of claim 9 , wherein the instructions, when executed by the one or more processors, further cause performing storing a plurality of key-value pairs for a plurality of compressed data blocks, wherein each key-value pair of the plurality of key-value pairs is unique at least amongst all key-value pairs of the plurality of key-value pairs.
12. The system of claim 11 , wherein the plurality of key-value pairs comprises at least one million unique keys.
13. The system of claim 9 , wherein creating the parse item comprises extracting the one or more parse tokens from one or more of the logical data entities.
14. The system of claim 9 , wherein the parse item further comprises snippet identifying information.
15. The system of claim 14 , wherein the snippet identifying information is a byte offset into an uncompressed block and a byte length.
16. The system of claim 9 , wherein the parse tokens identify byte-sequential portions of the data block.