Policy-based storage structure distribution
Policy-based storage and retrieval combined with a distribution algorithm results in automatic and even distribution of policy-based storage structures across a set of nodes and dynamic, automated homing or ownership of policy-based storage structures. Large numbers of policy-based storage structures may be distributed without manual administration, allowing for rapid creation and destruction of storage structures. The overall load may be distributed and balanced across the server pool. Multiple entries having the same key value in a database- or table-like structure allow for distribution of policy-based storage and retrieval by key value and for queue semantics to be utilized for microqueues in the large database- or table-like structure.
1. A method for distributing policy-based data structures over a system of one or more servers installed on one or more computing devices, the method comprising:
determining at least one policy-based data structure from a space, the determining comprising:
identifying data entries in the space;
determining key values of the data entries; and
mapping the at least one policy-based data structure to one of a node, node cluster, server, and server cluster in the system of one or more servers, the mapping comprising using a distribution algorithm to create node identifiers using node values and microqueues using the key values;
wherein the mapping the at least one policy-based data structure comprises distributing microqueues over the system of one or more servers;
wherein at least one microqueue comprises at least two data entries with each data entry having a same key value;
wherein a microqueue identifier maps to the same key value and to a node in the system of one or more servers.
2. The method of claim 1 , wherein the mapping comprises using a consistent hashing algorithm.
3. The method of claim 1 , wherein the mapping comprises:
using a consistent hashing algorithm to map nodes in the system of one or more servers to a scale;
using the consistent hashing algorithm to map the at least one policy-based data structure to the scale;
determining whether the at least one policy-based data structure is assigned to a node in the system of one or more servers;
in response to determining the at least one policy-based data structure is not assigned to a node in the system of one or more servers, assigning the at least one policy-based data structure to a closest node in the system of one or more servers, the closest node being the node mapped closest to the policy-based data structure on the scale.
4. The method of claim 3 , wherein the using a consistent hashing algorithm to map nodes in the system of one or more servers creates a distributed fault-tolerant system of one or more servers.
5. The method of claim 3 , wherein the using a consistent hashing algorithm to map nodes in the system of one or more servers to a scale results in a substantially even distribution of nodes.
6. The method of claim 3 , wherein the using a consistent hashing algorithm to map nodes in the system of one or more servers to a scale results in a substantially uneven distribution of nodes.
7. The method of claim 1 , wherein the determining at least one policy-based data structure in the space comprises determining a plurality of policy-based data structures by grouping data entries in the space based on each data entry's key value, and wherein the mapping the plurality of policy-based data structures comprises:
using a consistent hashing algorithm to map nodes in the system of one or more servers to a scale;
using the consistent hashing algorithm to map the plurality of policy-based data structures to the scale, wherein each of the plurality of policy-based data structures comprises a grouping of data entries having a same key value, and wherein the mapping the plurality of policy-based data structures to the scale comprises mapping the key values of each of the plurality of policy-based data structures to the scale; and
assigning the plurality of policy-based data structures to the mapped nodes by assigning each of the policy-based data structures to a node mapped closest to it on the scale.
8. The method of claim 1 , wherein the mapping the at least one policy-based data structure results in a substantially even distribution of policy-based data structures over a set of nodes or servers.
9. A method for distributing policy-based data structures over a system of one or more servers installed on one or more computing devices, the method comprising:
determining at least one policy-based data structure from a space, the determining comprising:
identifying at least one data entry in the space;
determining a key value of the at least one data entry;
mapping the at least one policy-based data structure to one of a node, node cluster, server, and server cluster in the system of one or more servers, the mapping comprising using a distribution algorithm considering the key value;
wherein the mapping comprises:
using a consistent hashing algorithm to map nodes in the system of one or more servers to a scale;
using the consistent hashing algorithm to map the at least one policy-based data structure to the scale;
determining whether the at least one policy-based data structure is assigned to a node in the system of one or more servers;
in response to determining the at least one policy-based data structure is not assigned to a node in the system of one or more servers, assigning the at least one policy-based data structure to a closest node in the system of one or more servers, the closest node being the node mapped closest to the policy-based data structure on the scale.