Methods for enhancing rapid data analysis

1. A method for identifying anomalous data in a computer database comprising:

receiving a set of data;

storing the set of data in a first set of database shards of the computer database; wherein the first set of database shards is partitioned, according to a set of shard partitioning rules, by a first field;

identifying anomalous data from the set of data, wherein the anomalous data is a subset of the set of data, by:

monitoring shard key density, shard key range, and shard size of the first set of database shards to identify at last one of anomalous shard key density, anomalous shard key range, and anomalous shard size;

in response to identification of at least one of an anomalous shard key density, an anomalous shard key range, and an anomalous shard size for a first database shard of the first set of database shards, detecting that the first database shard may contain anomalous data; and

identifying data of the first database shard as anomalous data; and

restructuring the anomalous data in response to identification of the anomalous data.

2. The method of claim 1 , wherein the anomalous shard key density is a shard key density that is lower than a threshold value.

3. The method of claim 1 , wherein the anomalous shard key range is a shard key range that is lower than a threshold value.

4. The method of claim 1 , wherein the anomalous shard size is a shard size that is smaller than a first threshold value or larger than a second threshold value.

5. The method of claim 1 , wherein restructuring the anomalous data comprises using a partitioning algorithm to partition the anomalous data into subsets based on values of a second field of the set of data.

6. The method of claim 1 , wherein in response to identification of an anomalous shard key density, data of the first database shard is identified as anomalous data, and the anomalous data is restructured in response to the identification of the anomalous data.

7. A method for data analysis, enhanced by identification of anomalous data in a computer database, the method comprising:

receiving a set of data;

storing the set of data as database shards of the computer database; wherein the database shards are partitioned, according to a set of shard partitioning rules, by a first field;

receiving and interpreting a query; wherein interpreting the query comprises identifying a first set of the database shards containing data relevant to the query;

identifying anomalous data from the set of data, wherein the anomalous data is a subset of the set of data, by:

monitoring shard key density, shard key range, and shard size of the first set of database shards to identify at last one of anomalous shard key density, anomalous shard key range, and anomalous shard size;

in response to identification of at least one of an anomalous shard key density, an anomalous shard key range, and an anomalous shard size for a first database shard of the first set of database shards, detecting that the first database shard may contain anomalous data; and

identifying data of the first database shard as anomalous data; and

collecting a first data sample from the first set of the database shards;

calculating a result to the query based on analysis of the first data sample;

wherein the analysis of the first data sample weights the anomalous data differently than other data of the first data sample.

8. The method of claim 7 , wherein the anomalous shard key density is a shard key density that is lower than a threshold value.

9. The method of claim 7 , wherein the anomalous shard key range is a shard key range that is lower than a threshold value.

10. The method of claim 7 , wherein the anomalous shard size is a shard size that is smaller than a first threshold value or larger than a second threshold value.

11. The method of claim 7 , further comprising flagging the anomalous data using metadata.

12. The method of claim 7 , further comprising restructuring the anomalous data in response to identification of the anomalous data.

13. The method of claim 12 , wherein restructuring the anomalous data comprises using a partitioning algorithm to partition the anomalous data into subsets based on values of a second field of the set of data.

14. The method of claim 7 , wherein calculating a result to the query based on analysis of the first data sample comprises:

performing intermediate calculations on the first data sample to identify intermediate query results, wherein the first data sample includes at least a portion of the anomalous data, and wherein performing intermediate calculations comprises weighting the anomalous data differently than other data of the first data sample; and

performing a full scan of all shards of the first set that contain data related to the intermediary results to calculate the result to the query.

15. The method of claim 7 , wherein collecting the first data sample comprises performing data sampling to ignore the identified anomalous data.