Methods for enhancing rapid data analysis

1. A method for enhancing rapid data analysis by efficiently identifying anomalous data in a computer database comprising:

receiving a set of data;

storing the set of data in a first set of database shards of the computer database; wherein the first set of database shards is partitioned into database shards, according to a set of shard partitioning rules, by a first field; and

identifying anomalous data from the set of data, wherein the set of anomalous data is a subset of the set of data, by:

monitoring a range of shard indices associated with a first database shard of the first set of database shards;

detecting that the range of shard indices is smaller than an expected range by a threshold value; and

identifying data of the first database shard as anomalous data.

2. The method of claim 1 , further comprising flagging the anomalous data using metadata.

3. The method of claim 1 , further comprising restructuring the anomalous data in response to identification of the anomalous data.

4. The method of claim 3 , wherein restructuring the anomalous data comprises using a partitioning algorithm to partition the anomalous data into subsets based on values of a second field of the set of data.

5. The method of claim 1 , further comprising generating an aggregate of the anomalous data in response to identification of the anomalous data.

6. A method for enhancing rapid data analysis by efficiently identifying anomalous data in a computer database comprising:

receiving a set of data;

storing the set of data as database shards of the computer database; wherein the database shards are partitioned into database shards, according to a set of shard partitioning rules, by a first field;

receiving and interpreting a query; wherein interpreting the query comprises identifying a first set of the database shards containing data relevant to the query;

collecting a first data sample from the first set of the database shards;

identifying anomalous data in the first data sample;

calculating a result to the query based on analysis of the first data sample; wherein the analysis of the first data sample ignores the anomalous data.

7. The method of claim 6 , further comprising restructuring the anomalous data in response to identification of the anomalous data.

8. The method of claim 7 , wherein restructuring the anomalous data comprises using a partitioning algorithm to partition the anomalous data into subsets based on values of a second field of the set of data.

9. The method of claim 6 , wherein identifying anomalous data comprises identifying anomalous data by, during collection of the first data sample, determining that a data element count associated with a value of a second field of the set of data exceeds a threshold data element count.

10. The method of claim 9 , wherein the threshold data element count is set according to a statistical distribution of data element counts, each data element count associated with a value of a range of values of a second field of the set of data, across the range of values.

11. The method of claim 10 , wherein the second field is the first field.

12. The method of claim 6 , wherein the query contains custom query code; wherein interpreting the query comprises pre-processing the custom query code by converting the custom query code from a foreign language to a native query language.

13. The method of claim 12 , wherein the foreign language is a natural language; wherein interpreting the query comprises interpreting the natural language using lexical analysis.

14. The method of claim 12 , further comprising compiling and executing the custom query code during execution of the query.

15. The method of claim 6 , further comprising generating an aggregate of the anomalous data in response to identification of the anomalous data.

16. A method for enhancing rapid data analysis by efficiently identifying anomalous data in a computer database comprising:

receiving a set of data;

storing the set of data in a first set of database shards of the computer database; wherein the first set of database shards is partitioned into database shards, according to a set of shard partitioning rules, by a first field; and

identifying anomalous data from the set of data, wherein the set of anomalous data is a subset of the set of data, by:

analyzing the set of data to determine a statistical distribution of data element counts, each data element count associated with a value of a range of values of a second field of the set of data, across the range of values;

determining that a value of the range of values is a statistical outlier based on the number of data element counts associated with the value; and

identifying data associated with the value as anomalous data.

17. The method of claim 16 , further comprising flagging the anomalous data using metadata.

18. The method of claim 16 , further comprising restructuring the anomalous data in response to identification of the anomalous data.

19. The method of claim 18 , wherein restructuring the anomalous data comprises using a partitioning algorithm to partition the anomalous data into subsets based on values of a third field of the set of data.

20. The method of claim 16 , further comprising generating an aggregate of the anomalous data in response to identification of the anomalous data.

21. The method of claim 16 , wherein the second field is the first field.