SYSTEMS AND METHODS FOR RAPID DATA ANALYSIS
A method for rapid data analysis includes receiving and interpreting a first query operating on a first dataset partitioned into shards by a first field; collecting a first data sample from a first set of data shards; calculating a first result to the first query based on analysis of the first data sample; and partitioning a second dataset into shards by a second field based on the first result.
1 . A system, comprising:
an event database, wherein data in the event database is partitioned into a plurality of data shards, wherein each data shard of the plurality of data shards stores a representative sample of a population; and
a query engine configured to:
receive a query;
determine a first data sample from a set of data shards of the plurality of data shards based on the query;
determine intermediate calculations based on the first data sample;
determine a final data sample based on the intermediate calculations; and
determine a query result based on the final data sample.
2 . The system of claim 1 , wherein the data in the event database is partitioned using vertical partitioning.
3 . The system of claim 2 , wherein the data in the event database is stored in a columnar format, wherein vertical partitioning comprises partitioning the data in the event database into vertical partitions, wherein each vertical partition comprises data in a subset of columns.
4 . The system of claim 2 , wherein data in the event database is further partitioned using horizontal partitioning.
5 . The system of claim 4 , wherein horizontal partitioning comprises partitioning each vertical partition by time information.
6 . The system of claim 1 , wherein data in the event database is partitioned based on a first ruleset and a second ruleset.
7 . The system of claim 6 , wherein the first ruleset comprises a vertical partitioning rule, wherein the second ruleset comprises a horizontal partitioning rule for each vertical partition.
8 . The system of claim 1 , wherein each data shard comprises data from an intersection of a subset of rows in the data in the event database and a subset of columns of the data in the data in the event database.
9 . The system of claim 1 , wherein the query engine is further configured to determine confidence information for an accuracy of the query result.
10 . The system of claim 1 , wherein the query engine is further configured to return the query result.
11 . A system, comprising:
a data manager configured to partition data into a plurality of data shards, wherein a distribution of the data across the plurality of data shards is statistically significant; and
a query engine configured to:
receive a query;
determine a first data sample from a set of data shards of the plurality of data shards based on the query;
determine intermediate calculations based on the first data sample;
determine a final data sample based on the intermediate calculations; and
determine a query result based on the final data sample.
12 . The system of claim 11 , wherein the data manager is configured to partition the data into the plurality of data shards using vertical partitioning and horizontal partitioning.
13 . The system of claim 12 , wherein vertical partitioning comprises partitioning the data into vertical partitions, wherein each vertical partition comprises data for a subset of vertical fields of the data.
14 . The system of claim 13 , wherein each subset of vertical fields comprises a time field.
15 . The system of claim 13 , wherein horizontal partitioning comprises partitioning each vertical partition into a predetermined number of horizontal partitions.
16 . The system of claim 11 , wherein each data shard comprises data from an intersection of a subset of rows in the data and a subset of columns of the data.
17 . The system of claim 11 , further comprising a distributed computing system, wherein the plurality of data shards are stored across the distributed computing system.
18 . The system of claim 11 , wherein the data manager is further configured to encode the plurality of data shards.
19 . The system of claim 11 , wherein the query is determined by a user using a graphical user interface.
20 . The system of claim 11 , wherein the query engine is further configured to return the query result.