Randomized data distribution in highly parallel database management system
A massively parallel database management system includes an index store and a payload store including a set of storage systems of different temperatures. Both the index store and the storage system each include a list of clusters. Each cluster includes a set of nodes with storage devices forming a group of segments. Nodes and clusters are connected over high speed links. Each cluster receives data and splits the data into data rows based on a predetermined size. The data rows are randomly and evenly distributed between all nodes of the cluster.
1. A payload store within a database management system, the payload store comprising:
a set of coding clusters, wherein a first coding cluster of the set of coding clusters includes a first set of nodes and a second coding cluster of the set of coding clusters includes a second set of nodes, wherein the first set of nodes and the second set of nodes have an equal number of nodes, and wherein a first node of the first set of nodes includes:
a memory;
a network interface;
a storage device; and
a processing unit operatively coupled to the memory, the network interface, and the storage device,
wherein the processing unit is operable to:
receive, via the network interface, a first portion of data, wherein the first portion is to be stored in storage devices of the first coding cluster;
split the first portion into a first set of data rows, wherein the first set of data rows includes a plurality of subsets of data rows and a last subset of data rows; and
randomly assign the plurality of subsets of data rows and the last subset of data rows between the first set of nodes, wherein a first number of data rows of the plurality of subsets of data rows is assigned to the first node and a second number of data rows of the plurality of subsets of data rows is assigned to a second node of the first set of nodes, and wherein the first number and the second number are equal;
store, by the first node, the first number of data rows in the storage device;
send, via the network interface, the second number of data rows to the second node for storage therein; and
send, by the first node and via the network interface, the first and second number of data rows to a parity node of the first set of nodes, wherein the parity node is operable to calculate parity information for the first and second number of data rows, and wherein the parity node stores the parity information in a storage device of the parity node.
2. The payload store of claim 1 , wherein the processing unit is further operable to:
compress the first portion of data to produce a compressed first portion; and
split the compressed first portion into the first set of data rows.
3. The payload store of claim 1 , wherein each subset of the plurality of subsets of data rows has a first size and the last subset of data rows has a size that is equal to or less than the first size.
4. The payload store of claim 1 , wherein a third node of the second set of nodes includes:
a second memory;
a second network interface;
a second storage device; and
a second processing unit operatively coupled to the second memory, the second network interface, and the second storage device, wherein the second processing unit is operable to:
receive, via the second network interface, a second portion of the data, wherein the second portion is to be stored in second storage devices of the second coding cluster;
split the second portion of the data into a second set of data rows;
randomly assign the second set of data rows between the second set of nodes, wherein a third number of data rows of the second set of data rows are assigned to the third node; and
store, by the second node, the third number of data rows.
5. The payload store of claim 1 , wherein processing unit is further operable to randomly assign the plurality of subsets of data rows and the last subset of data rows by:
randomly assigning a third number of data rows of the first set of data rows to a third node of the first set of nodes, wherein the third number of data rows includes the last subset of data rows of the first set of data rows, and wherein the third number is equal to or less than the first and second numbers.
6. The payload store of claim 1 , wherein the second node of the first set of nodes includes:
a second memory;
a second network interface;
a second storage device; and
a second processing unit operatively coupled to the second memory, the second network interface, and the second storage device, wherein the second processing unit is operable to:
receive, via the second network interface, the second number of data rows from the first node; and
store the second number of data rows in the second storage device.