Distributed database with modular blocks and associated log files
A system includes a master node and worker nodes controlled by the master node. Each worker node stores 25 or more modular blocks of a distributed database, where each modular block has a size of 5G bytes or less and has an associated log file.
1. A system, comprising:
a master node; and
a plurality of worker nodes controlled by the master node, wherein each worker node stores 25 or more modular blocks of distributed database partitions, wherein each modular block holds a segment of data associated with the distributed database partitions, and wherein each modular block of the distributed database partitions has a size of 5G bytes or less and has an associated log file to record changes associated with the distributed database partition;
wherein the master node and the plurality of worker nodes are commodity hardware servers that communicate over a distributed network utilizing standard network connections and protocols; and
wherein the master node collects statistics about data in the modular blocks and uses the statistics to optimize a database query.
2. The system of claim 1 wherein the master node receives the database query and partitions the database query to run in parallel across the plurality of worker nodes.
3. The system of claim 1 wherein the master node collects statistics selected from modular block location, size, associated log file and update time.
4. The system of claim 3 wherein the master node communicates with the plurality of worker nodes to update the statistics.
5. The system of claim 1 wherein the master node maintains availability information for the plurality of worker nodes.
6. The system of claim 1 wherein the master node maintains distribution information for the modular blocks across the plurality of worker nodes.
7. The system of claim 5 wherein the master node determines the placement of new modular blocks using the availability information for the plurality of worker nodes.
8. The system of claim 6 wherein the master node determines the placement of new modular blocks based upon the distribution information for the modular blocks.
9. The system of claim 1 wherein the master node redistributes modular blocks across the plurality of worker nodes.
10. The system of claim 5 wherein the master node redistributes modular blocks based upon the availability information.
11. The system of claim 6 wherein the master node redistributes modular blocks based upon the distribution information for the modular blocks.
12. The system of claim 1 wherein the master node controls the replication of modular blocks across the plurality of worker nodes.
13. The system of claim 12 wherein the master node controls the replication of modular blocks based upon availability information for the plurality of worker nodes.
14. The system of claim 12 wherein the master node controls the replication of the modular blocks based upon distribution information for the modular blocks.
15. The system of claim 12 wherein replication is a configurable parameter.
16. The system of claim 1 wherein the modular blocks have a size configurable to any value less than 5 GB.
17. The system of claim 1 wherein the master node produces a new modular block when a current modular block exceeds a maximum size.
18. The system of claim 1 wherein the master node produces a new modular block with half of the information from a current modular block when the current modular block exceeds a maximum size.