Methods for enhancing rapid data analysis
A method for enhancing rapid data analysis includes receiving a set of data; storing the set of data in a first set of data shards sharded by a first field; and identifying anomalous data from the set of data by monitoring a range of shard indices associated with a first shard of the first set of data shards, detecting that the range of shard indices is smaller than an expected range by a threshold value, and identifying data of the first shard as anomalous data.
1. An anomalous data identification system, comprising:
a non-transitory computer readable medium storing instructions that, when executed by a processing system, cause the processing system to:
receive a dataset;
identify anomalous data comprising a subset of the dataset;
shard the dataset into database shards according to a set of partitioning rules while accounting for the anomalous data;
compress the anomalous data in a different manner than non-anomalous data;
receive a query;
identify a set of shards containing data relevant to the query;
collect a data sample from the set of shards; and
calculate a result to the query based on an analysis of the data sample.
2. The system of claim 1 , wherein the analysis of the data sample excludes the anomalous data.
3. The system of claim 1 , wherein the analysis of the data sample is based on weights assigned to the anomalous data and non-anomalous data, wherein the weights are different for the anomalous data and the non-anomalous data.
4. The system of claim 1 , wherein identifying the anomalous data comprises flagging the anomalous data with metadata.
5. The system of claim 4 , wherein the metadata comprises handling information.
6. The system of claim 1 , wherein the processing system is further configured to physically structure the anomalous data on disk or in memory differently from non-anomalous data.
7. The system of claim 1 , wherein the anomalous data is evenly distributed across the database shards.
8. The system of claim 1 , wherein compressing the anomalous data comprises using a different string dictionary than the non-anomalous data.
9. The system of claim 1 , wherein the processing system is further configured to store the anomalous data in different manner than non-anomalous data.
10. The system of claim 1 , wherein the anomalous data is identified using a manually configured ruleset.
11. The system of claim 1 , wherein the dataset is stored in persistent memory; and wherein the persistent memory comprises a plurality of contiguous blocks, wherein the database shards are stored in the plurality of contiguous blocks.
12. The system of claim 11 , wherein a block size of each block is defined by a threshold that represents a comparison between a cost of scanning a current block compared to a cost of scanning a next block of the plurality of contiguous blocks.
13. The system of claim 1 , wherein at least one of the non-anomalous data or the anomalous data is compressed lossless.
14. A system for anomalous data identification, comprising:
a database comprising a set of database shards collectively storing a dataset, wherein the dataset is partitioned into the database shards according to a set of partitioning rules; and
a processing system configured to identify anomalous data comprising a subset of the dataset by:
analyzing the dataset to determine a statistical distribution of data element counts;
identifying the anomalous data based on the statistical distribution;
structuring the anomalous data; and
compressing the anomalous data in a different manner than non-anomalous data.
15. The system of claim 14 , wherein identifying the anomalous data comprises tagging the anomalous data with metadata.
16. The system of claim 14 , wherein structuring the anomalous data comprises placing the anomalous data on slower disks compared to non-anomalous data.
17. The system of claim 14 , wherein the anomalous data is structured by partitioning the anomalous data based on a set of data fields of the anomalous data.
18. The system of claim 14 , wherein the dataset is stored in persistent memory; and wherein the persistent memory comprises a plurality of contiguous blocks, wherein the database shards are stored in the plurality of contiguous blocks.
19. The system of claim 14 , wherein the anomalous data is identified before the dataset is partitioned.
20. An anomalous data identification system, comprising:
a non-transitory computer readable medium storing instructions that, when executed by a processing system, cause the processing system to:
receive a dataset;
store the dataset as database shards, wherein the database shards are partitioned according to a set of partitioning rules;
receive a query;
identify a set of shards containing data relevant to the query;
collecting a data sample from the set of shards;
identify anomalous data comprising a subset of the data sample; and
calculate a result to the query based on an analysis of the data sample that treats the subset of the data sample differently from a remainder of the data sample, wherein the subset of the data sample is compressed in a different manner than the remainder of the data sample.