Methods for enhancing rapid data analysis
A method for enhancing rapid data analysis includes receiving a set of data; storing the set of data in a first set of data shards sharded by a first field; and identifying anomalous data from the set of data by monitoring a range of shard indices associated with a first shard of the first set of data shards, detecting that the range of shard indices is smaller than an expected range by a threshold value, and identifying data of the first shard as anomalous data.
1 . A system, comprising:
a database, wherein data in the database is partitioned into a plurality of data shards; and
a processing subsystem configured to:
receive a query;
identify a set of relevant data shards from the plurality of data shards containing data relevant to the query;
determine a data sample, by:
selecting a subset of the set of relevant data shards according to a set of sampling parameters; and
collecting the data sample from the subset of the set of relevant data shards; and
calculate a result based on an analysis of the data sample.
2 . The system of claim 1 , wherein the subset of the set of relevant data shards comprises a representative sample of a broader population.
3 . The system of claim 1 , wherein each data shard in the plurality of data shards comprises data from an intersection of a subset of rows in the data in the database and a subset of columns of the data in the database.
4 . The system of claim 1 , further comprising a distributed computing system comprising a plurality of data nodes storing the plurality of data shards.
5 . The system of claim 4 , wherein the plurality of data shards are distributed across the plurality of data nodes such that each data node stores data representative of a broader population.
6 . The system of claim 1 , wherein the processing subsystem is further configured to determine a confidence parameter for an accuracy of the result, wherein the set of sampling parameters are determined based on a target confidence parameter.
7 . The system of claim 1 , wherein the result comprises an intermediate calculation for a first pass, wherein the processing subsystem is configured to perform a multipass method comprising at least the first pass and a final pass, the final pass comprising:
determining a final data sample based on the intermediate calculation and a final set of sampling parameters; and
determining a final result based on the final data sample, wherein the processing system is configured to return the final result.
8 . The system of claim 7 , wherein the final set of sampling parameters is different from the set of sampling parameters.
9 . The system of claim 1 , wherein the processing subsystem is further configured to: identify anomalous data in the database, wherein the analysis of the data sample is based on weights assigned to the anomalous data and non-anomalous data, wherein the weights are different for the anomalous data and the non-anomalous data.
10 . The system of claim 1 , wherein the processing subsystem is further configured to: identify data shards in the plurality of data shards containing anomalous data based on an analysis of a sharding structure of the plurality of data shards; and flag the anomalous data with metadata, wherein the data sample is determined based on the metadata.
11 . A system, comprising:
a processing subsystem configured to:
partition a dataset into a plurality of data shards, wherein each data shard of the plurality of data shards stores a representative sample of a population;
compress each data shard of the plurality of data shards;
receive a query;
collect a data sample from a subset of the plurality of data shards; and
calculate a result based on an analysis of the data sample.
12 . The system of claim 11 , wherein the processing subsystem is configured to partition the dataset into the plurality of data shards using vertical partitioning and horizontal partitioning.
13 . The system of claim 12 , wherein vertical partitioning comprises partitioning the dataset into vertical partitions, wherein each vertical partition comprises data for a subset of vertical fields of the dataset, wherein horizontal partitioning comprises partitioning each vertical partition into a predetermined number of horizontal partitions.
14 . The system of claim 13 , wherein each subset of vertical fields comprises a time field.
15 . The system of claim 11 , wherein the processing subsystem is further configured to:
identify data shards of the plurality of data shards containing anomalous data based on an analysis of a sharding structure of the plurality of data shards; and
flag the anomalous data with metadata, wherein the data sample is collected based on the metadata.
16 . The system of claim 15 , wherein analyzing the sharding structure of the plurality of data shards comprises identifying shard keys that are statistical outliers based on a number of entries for each shard key.
17 . The system of claim 11 , wherein the processing subsystem is further configured to: identify anomalous data in the dataset base, wherein the analysis of the data sample is based on weights assigned to the anomalous data and non-anomalous data, wherein the weights are different for the anomalous data and the non-anomalous data.
18 . The system of claim 11 , further comprising a distributed computing system comprising a plurality of nodes, wherein the processing subsystem is further configured to encode the plurality of data shards, and store the encoded plurality of data shards across the plurality of nodes.
19 . The system of claim 11 , wherein the query is determined by a user using a graphical user interface.
20 . The system of claim 11 , wherein the processing subsystem is further configured to return the result of the query.