Systems and methods for rapid data analysis
A method for rapid data analysis includes receiving and interpreting a first query operating on a first dataset partitioned into shards by a first field; collecting a first data sample from a first set of data shards; calculating a first result to the first query based on analysis of the first data sample; and partitioning a second dataset into shards by a second field based on the first result.
1. A system, comprising:
a distributed computing system comprising a plurality of data nodes;
an event database stored in the distributed computing system, wherein data in the event database is horizontally and vertically partitioned into a plurality of data shards that are stored across the plurality of data nodes; and
a query engine, configured to: receive a query on the event database; evaluate the query using a multiple pass method comprising at least a first pass and a final pass; and return a query result; wherein evaluating the query using the multiple pass method comprises:
determining a first set of data shards of the plurality of data shards from which to obtain a first data sample in the first pass based on the query;
determining intermediate query results from the first data sample;
determining a final set of data shards of the plurality of data shards from which to obtain a final data sample based on the intermediate query results in the final pass; and
determining the query result based on the final data sample.
2. The system of claim 1 , wherein the plurality of data shards are partitioned by time information.
3. The system of claim 1 , wherein the plurality of data shards are automatically partitioned based on a ruleset.
4. The system of claim 3 , wherein the ruleset further comprises vertical partitions, each comprising a fixed number of fields and a fixed number of horizontal partitions.
5. The system of claim 1 , wherein the event database comprises datasets stored in a columnar format.
6. The system of claim 1 , wherein each data shard stores a representative sample of a broader population.
7. The system of claim 1 , further comprising:
a string lookup database configured to store a mapping of strings to string identifiers, wherein data in the string lookup database is stored and distributed across the plurality of data nodes of the distributed computing system; and
a string translator configured to translate a received string into a string identifier by looking up the string identifier corresponding with the received string in the string lookup database.
8. The system of claim 1 , wherein the query engine is further configured to:
identify the set of data shards from the plurality of data shards containing data relevant to the query;
collect a data sample from the set of data shards; and
calculate the query result to the query based on analysis of the data sample.
9. The system of claim 8 , wherein identifying the set of data shards comprises identifying data node locations of the set of data shards using a configuration database, wherein the configuration database stores shard identifiers and associated data node locations.
10. The system of claim 9 , wherein the shard identifiers comprise user identifiers.
11. The system of claim 8 , wherein calculating the query result further comprises determining confidence information for an accuracy of the query result.
12. The system of claim 1 , wherein the plurality of data nodes are grouped into subsets.
13. The system of claim 1 , wherein the query engine processes a query by parsing SQL-like query strings into a query tree or by translating strings to integers.
14. A system, comprising:
a distributed computing system comprising a plurality of data nodes;
an event database stored in the distributed computing system, wherein data in the event database is horizontally and vertically partitioned into a plurality of data shards that are stored across the plurality of data nodes, wherein each data shard of the plurality of data shards stores a representative sample of a population; and
a query engine, configured to: receive a query on the event database; evaluate the query using a multiple pass method comprising at least a first pass and a final pass; and return a query result; wherein evaluating the query using the multiple pass method comprises:
determining a first set of data shards of the plurality of data shards from which to obtain a first data sample in the first pass based on the query;
determining intermediate query results from the first data sample;
determining a final set of data shards of the plurality of data shards from which to obtain a final data sample based on the intermediate query results in the final pass; and
determining the query result based on the final data sample.
15. The system of claim 14 , wherein the query engine is further configured to collect samples from the set of data shards of the plurality of data shards, wherein the query result is determined based on the samples.
16. The system of claim 14 , wherein the query is determined by a user using a graphical user interface.
17. The system of claim 14 , wherein the query result comprises at least one of raw event data, processed event data, or confidence information.
18. The system of claim 14 , wherein the data shards are distributed across the plurality of data nodes such that each data node stores data representative of a broader population.
19. The system of claim 14 , wherein the plurality of data shards comprise explicit and implicit data.
20. The system of claim 14 , wherein the plurality of data nodes are grouped into subsets.