Systems and methods for rapid data analysis

1. A system for rapid data analysis, comprising a set of distributed non-volatile memory collectively storing a dataset of data from a plurality of data sources, wherein each non-volatile memory stores a set of data shards, wherein each data shard stores a subset of the dataset, the system configured to:

receive a query;

interpret the query, comprising identifying a first set of data shards containing data relevant to the query;

determine a first data sample from the first set of data shards;

calculate an intermediate result to the query based on the first data sample;

determine a set of query-relevant data sources based on the intermediate result;

identify a second set of data shards from the set of query-relevant data sources;

determine a second data sample from the second set of data shards; and

calculate a final result to the query based on the second data sample.

2. The system of claim 1 , wherein the second set of data shards contains data distinct from the data contained in first set of data shards.

3. The system of claim 1 , further configured to determine confidence information for an accuracy of the final result.

4. The system of claim 1 , wherein interpreting the query further comprises translating a string of the query to a set of integers.

5. The system of claim 1 , wherein the query includes a time range and an event data source.

6. The system of claim 1 , wherein the first set of data shards and the second set of data shards comprise explicit and implicit data.

7. The system of claim 1 , wherein the first data sample is identified after receipt of the query.

8. The system of claim 1 , wherein the first set of data shards is distributed across the set of distributed non-volatile memory; and wherein the first data sample is identified by simultaneously performing a plurality of parallel reads on the set of distributed non-volatile memory.

9. A system for rapid data analysis comprising a set of non-volatile memory collectively storing a dataset, wherein each non-volatile memory within the set stores a set of data shards, wherein each data shard stores a distinct subset of the dataset, the system configured to:

receive a query;

identify a first set of data shards containing data relevant to the query;

identify a first data sample from the first set of data shards;

determine a first intermediate result to the query based on the first data sample;

perform a non-zero number of intermediate searches, wherein each intermediate search comprises:

identifying an additional set of data shards based on at least one of the first intermediate result and additional intermediate results;

identifying additional data samples from the additional set of data shards; and

determining additional intermediate results based on the additional data samples; and

determine a final result to the query.

10. The system of claim 9 , further configured to determine confidence information for each additional intermediate result; and wherein the non-zero number of intermediate searches are performed until the confidence information exceeds a confidence threshold.

11. The system of claim 9 , wherein the query comprises a variable that represents a size of the first data sample.

12. A system, comprising:

an event database comprising a set of non-volatile memory collectively storing a dataset within a plurality of data shards, wherein each data shard stores a subset of the dataset, the comprising event data and associated sequence identifiers, wherein the event data is organized based on the associated sequence identifiers; and

a query engine, configured to:

receive a query;

calculate an intermediate result to the query based on analysis of a first data sample, wherein the first data sample is determined from a first set of data shards of the plurality of data shards;

identify a set of query-relevant data sources based on the intermediate result;

identify a second set of data shards from the plurality of data shards using the set of query-relevant data sources;

identify a second data sample from the second set of data shards; and

calculate a final result to the query based on the second data sample.

13. The system of claim 12 , wherein each sequence identifier is a timestamp.

14. The system of claim 12 , further comprising a string translator configured to translate strings into respective string identifiers; wherein the query engine is further configured to determine a set of string identifiers associated with the query using the string translator.

15. The system of claim 14 , wherein the set of string identifiers are numerical identifiers.

16. The system of claim 14 , further comprising a string lookup database configured to store a mapping of strings to string identifiers, wherein the mapping enables prefix matching.

17. The system of claim 12 , further comprising a configuration database configured to store shard mapping data; wherein the second set of data shards is identified from the set of query-relevant data sources using the shard mapping data.

18. The system of claim 12 , wherein the query comprises a query field that does not correspond to a stored data field within the event database.

19. The system of claim 18 , wherein the intermediate result is calculated by:

determining a derived field based on the query field based on the stored event data of the event database; and

determining the intermediate result based on the derived field.

20. The system of claim 12 , wherein the event database comprises a columnar dataset.