Soft capacity constraints for storage assignment in a distributed environment
A system and method for balancing data storage among a plurality of groups of computing devices, each group comprising one or more respective computing devices, each group having an available storage capacity. The method may involve, for each group of computing devices, determining an amount of used storage at the group of computing devices exceeding a predefined first threshold value that is less than the available storage capacity and calculating a storage cost based on the determined amount of used storage exceeding the predefined first threshold value, determining a total storage cost of the plurality of groups of computing devices based on a sum of the calculated storage costs, determining a transfer of one or more projects between the groups of computing devices that reduces the total storage and directing the plurality of groups of computing devices to execute the determined transfer.
1 . A method of balancing data storage among a plurality of computing nodes, each computing node comprising one or more respective computing devices, each computing node having an available capacity, the method comprising:
for a given computing node of computing devices:
determining for the given computing node, by one or more processors, a measure of used capacity exceeding a predefined first threshold value that is less than the available capacity;
calculating, by the one or more processors, a cost based on the determined measure of used capacity exceeding the predefined first threshold value; and
determining, by the one or more processors, a transfer of one or more projects from the given computing node that reduces the calculated cost, wherein determining the transfer is performed using a solver; and
outputting, by the one or more processors, one or more control signals to the given computing node to execute the determined transfer of one or more projects to balance the data storage among the plurality of computing nodes.
2 . The method of claim 1 , wherein the available capacity of each computing node further includes a hard buffer to which storage of new projects at the computing node is restricted, wherein the first threshold value is greater than a difference between the available capacity and a size of the hard buffer.
3 . The method of claim 1 , wherein calculating the cost comprises multiplying the determined measure of used capacity exceeding the predefined first threshold value by a predefined first constant value.
4 . The method of claim 3 , further comprising, for the given computing node, determining for the given computing node, by the one or more processors, a measure of used capacity exceeding a predefined second threshold value that is less than the available capacity, the second threshold value being greater than the first threshold value, wherein calculating the cost comprises multiplying the determined measure of used capacity exceeding the predefined second threshold value by a predefined second constant value.
5 . The method of claim 4 , wherein the predefined second constant value is greater than or equal to the predefined first constant value.
6 . The method of claim 3 , further comprising, for the given computing node, determining for the given computing node, by the one or more processors, a measure of used capacity exceeding n predefined threshold values, each predefined threshold value being less than the available capacity, wherein n is an integer equal to or greater than two, and wherein calculating the cost comprises multiplying the determined measure of used capacity by one of n predefined constant values depending on how many of the n predefined threshold values the measure of used capacity at the given computing node exceeds.
7 . The method of claim 1 , wherein calculating the cost comprises inputting the determined measure of used capacity exceeding the predefined first threshold value to a non-linear function.
8 . The method of claim 1 , further comprising:
determining, by the one or more processors, one or more secondary cost factors associated with transferring the one or more projects between the plurality of computing nodes;
assigning, by the one or more processors, a first weighting value to the calculated cost and a respective weighting value to each of the one or more secondary factors; and
calculating, by the one or more processors, a sum of the weighted calculated cost and the weighted secondary cost factors,
wherein the determined transfer of the one or more projects between the computing nodes reduces the calculated sum.
9 . The method of claim 8 , wherein the one or more secondary cost factors includes a network bandwidth cost of transferring the one or more projects between the computing nodes.
10 . The method of claim 8 , wherein the one or more secondary cost factors includes a network communication cost of communicating data between the computing nodes.
11 . A system of balancing data storage among a plurality of computing nodes, each computing node comprising one or more respective computing devices, each computing node having an available capacity, the system comprising:
one or more memories; and
one or more processors in communication with the one or more memories, the one or more processors configured to:
for a given computing node:
determine, for the given computing node, a measure of used capacity exceeding a predefined first threshold value that is less than the available capacity; and
calculate a cost based on the determined measure of used capacity exceeding the predefined first threshold value;
determine a transfer of one or more projects between the computing nodes that reduces a total cost, wherein the determination of the transfer is performed by the one or more processors using a solver; and
output control signals to the plurality of computing nodes to execute the determined transfer of one or more projects between the computing nodes to balance the data storage among the plurality of computing nodes.
12 . The system of claim 11 , wherein the available capacity of each computing node further includes a hard buffer to which storage of new projects at the computing node is restricted, wherein the first threshold value is greater than a difference between the available capacity and a size of the hard buffer.
13 . The system of claim 11 , wherein the one or more processors are configured to calculate the cost by multiplying the determined measure of used capacity exceeding the predefined first threshold value by a predefined first constant value.
14 . The system of claim 13 , wherein the one or more processors are configured to, for the given computing node, determine for the given computing node, an measure of used capacity exceeding a predefined second threshold value that is less than the available capacity, the second threshold value being greater than the first threshold value, wherein the one or more processors are configured to calculate the cost by multiplying the determined measure of used capacity exceeding the predefined second threshold value by a predefined second constant value.
15 . The system of claim 14 , wherein the predefined second constant value is greater than or equal to the predefined first constant value.
16 . The system of claim 13 , wherein the one or more processors are configured to, for the given computing node, determine for the given computing node, a measure of used capacity exceeding n predefined threshold values, each predefined threshold value being less than the available capacity, wherein n is an integer equal to or greater than two, and wherein the one or more processors are configured to calculate the cost by multiplying the determined measure of used capacity by one of n predefined constant values depending on how many of the n predefined threshold values the measure of used capacity at the given computing node exceeds.
17 . The system of claim 11 , wherein the one or more processors are configured to calculate the cost by inputting the determined measure of used capacity exceeding the predefined first threshold value to a non-linear function.
18 . The system of claim 11 , wherein the one or more processors are configured to:
determine one or more secondary cost factors associated with transferring the one or more projects between the plurality of computing nodes;
assign a first weighting value to the calculated cost and a respective weighting value to each of the one or more secondary factors; and
calculate a sum of the weighted calculated cost and the weighted secondary cost factors, wherein the determined transfer of the one or more projects between the computing nodes reduces the calculated sum.
19 . The system of claim 18 , wherein the one or more secondary cost factors includes a network bandwidth cost of transferring the one or more projects between the computing nodes.
20 . The system of claim 18 , wherein the one or more secondary cost factors includes a network communication cost of communicating data between the computing nodes.