File block placement in a distributed network

1. A method for a file block placement by predicting available bandwidth for a candidate flow on a link in a distributed network, the method comprising:

obtaining information about a plurality of flows carried by the link, the information about the plurality of flows carried by the link including a current bandwidth consumption for each flow of the plurality of flows carried by the link;

identifying whether each flow of the plurality of flows has a local constraint or a remote constraint, wherein the local constraint is a throughput constraint of the each flow of the link and the remote constraint is a throughput constraint of each flow of another physical link other than the link; and

computing the available bandwidth for the candidate flow based at least in part on the information about the plurality of flows carried by the link and the identification of whether each flow of the plurality of flows has a local constraint or a remote constraint to generate a predicted available bandwidth for the candidate flow;

selecting one of a plurality of file block placement options based at least in part on the predicted available bandwidth which meets a particular quality of service (QoS) requirement;

determining a set of storage nodes which correspond to the one of the plurality of file block placement options as selected; and

performing the file block placement to the set of storage nodes, wherein computing the available bandwidth for the candidate flow comprises:

subtracting a sum of the current bandwidth consumption of the flows carried by the link having a remote constraint from a capacity of the link; and

dividing the difference by the number of flows carried by the link plus one minus the number of flows carried by the link having a remote constraint.

2. The method of claim 1 , wherein identifying whether each flow of the plurality of flows has a local constraint or a remote constraint comprises:

comparing the current bandwidth consumption for each flow of the plurality of flows carried by the link to a maximum current bandwidth consumption of the plurality of flows carried by the link; and

identifying, in the case that the current bandwidth consumption for a flow of the plurality of flows carried by the link is approximately equal to the maximum current bandwidth consumption of the plurality of flows carried by the link, that flow as having a local constraint.

3. The method of claim 1 , further comprising determining a subset of active flows of the plurality of flows by comparing the current bandwidth consumption of each flow of the plurality of flows to a threshold.

4. The method of claim 1 , wherein identifying whether each flow of the plurality of flows has a local constraint or a remote constraint comprises:

determining an unused capacity of the link;

sorting the flows carried by the link having based on the associated current bandwidth consumption; and

identifying a set of the sorted flows having the largest associated current bandwidth consumption as having a local constraint.

5. The method of claim 4 , wherein computing the available bandwidth for the candidate flow comprises:

summing the unused capacity of the link and a total of the current bandwidth consumptions of the flows identified as having a local constraint; and

dividing the sum of the number of flows identified as having a local constraint plus one.

6. The method of claim 1 , further comprising determining a subset of active flows of the plurality of flows and predicting a bandwidth consumption for at least some of the active flows.

7. The method of claim 6 , wherein predicting the bandwidth consumption comprises:

predicting, for an active flow having a current bandwidth consumption less than the computed available bandwidth, the bandwidth consumption to be the current bandwidth consumption associated with that flow; and

predicting, for an active flow having a current bandwidth consumption greater than the computed available bandwidth, the bandwidth consumption to be the computed available bandwidth.

8. The method of claim 6 , further comprising:

generating a plurality of file block placement options for a file block, each file block placement option being associated with at least one of a plurality of data storage nodes in the distributed network, the file block placement options being based on a set of network parameters associated with the distributed network; and

selecting one of the plurality of file block placement options based at least in part on the predicted bandwidth consumptions.

9. The method of claim 8 , further comprising determining whether each of the plurality of file block placement options satisfies requirements associated with users of the distributed network.

10. The method of claim 9 , wherein the set of network parameters includes one or more constraints, and wherein the method further comprises, when none of the plurality of file block placement options satisfies the requirements associated with the users of the distributed network:

relaxing at least one of the constraints;

generating a second plurality of file block placement options for the file block using the relaxed at least one constraint;

predicting a bandwidth consumption for each active flow for a candidate file block placement option of the second plurality of file block placement options; and

selecting one of the second plurality of file block placement options based at least in part on the predicted bandwidth consumptions associated with the candidate file block placement option of the second plurality of file block placement options.

11. The method of claim 9 , further comprising, when none of the plurality of file block placement options satisfies the requirements associated with the users of the distributed network:

reducing the requirements associated with users of the distributed network by excluding at least one of the requirements associated with at least one of the users of the distributed network; and

determining whether each of the plurality of file block placement options satisfies the reduced requirements.

12. The method of claim 9 , further comprising, when none of the plurality of file block placement options satisfies the requirements associated with the users of the distributed network, delaying selecting one of the plurality of file block placement options.

13. The method of claim 9 , further comprising, when none of the plurality of file block placement options satisfies the requirements associated with the users of the distributed network:

predicting a bandwidth consumption for at least some of the active flows for a candidate file block placement option of the plurality of file block placement options taking into account suspension of one or more existing block replication processes;

selecting one of the plurality of file block placement options based at least in part on the predicted bandwidth consumptions taking into account the suspension of one or more existing block replication processes; and

suspending the one or more existing block replication processes.

14. A computing device for predicting available bandwidth for a file block placement of a candidate flow on a link in a distributed network, the computing device comprising:

a memory configured to store data and processing instructions; and

a processor configured to retrieve and execute the processing instructions stored in the memory to cause the processor to perform the steps of:

obtaining information about a plurality of flows carried by the link, the information about the plurality of flows carried by the link including a current bandwidth consumption for each flow of the plurality of flows carried by the link;

identifying whether each flow of the plurality of flows has a local constraint or a remote constraint, wherein the local constraint is a throughput constraint of the each flow of the link and the remote constraint is a throughput constraint of each flow of another physical link other than the link; and

computing the available bandwidth for the candidate flow based at least in part on the information about a plurality of flows carried by the link and the identification of whether each flow of the plurality of flows has a local constraint or a remote constraint to generate a predicted available bandwidth for the candidate flow;

selecting one of a plurality of file block placement options based at least in part on the predicted available bandwidth which meets a particular quality of service (QoS) requirement;

determining a set of storage nodes which correspond to the one of the plurality of file block placement options as selected; and

performing the file block placement to the set of storage nodes, wherein computing the available bandwidth for the candidate flow comprises:

subtracting a sum of the current bandwidth consumption of the flows carried by the link having a remote constraint from a capacity of the link; and

dividing the difference by the number of flows carried by the link plus one minus the number of flows carried by the link having a remote constraint.

15. The computing device of claim 14 , wherein identifying whether each flow of the plurality of flows has a local constraint or a remote constraint comprises:

comparing the current bandwidth consumption for each flow of the plurality of flows carried by the link to a maximum current bandwidth consumption of the plurality of flows carried by the link; and

identifying, in the case that the current bandwidth consumption for a flow of the plurality of flows carried by the link is approximately equal to the maximum current bandwidth consumption of the plurality of flows carried by the link, that flow as having a local constraint.

16. The computing device of claim 14 , wherein the processing instructions further cause the processor to perform the step of determining a subset of active flows of the plurality of flows by comparing the current bandwidth consumption of each flow of the plurality of flows to a threshold.

17. The computing device of claim 14 , wherein identifying whether each flow of the plurality of flows has a local constraint or a remote constraint comprises:

determining an unused capacity of the link;

sorting the flows carried by the link having based on the associated current bandwidth consumption; and

identifying a set of the sorted flows having the largest associated current bandwidth consumption as having a local constraint.

18. The computing device of claim 17 , wherein computing the available bandwidth for the candidate flow comprises:

summing the unused capacity of the link and a total of the current bandwidth consumptions of the flows identified as having a local constraint; and

dividing the sum of the number of flows identified as having a local constraint plus one.

19. The computing device of claim 14 , wherein the processing instructions further cause the processor to perform the steps of determining a subset of active flows of the plurality of flows and predicting a bandwidth consumption for at least some of the active flows.

20. The computing device of claim 19 , wherein predicting the bandwidth consumption comprises:

predicting, for an active flow having a current bandwidth consumption less than the computed available bandwidth, the bandwidth consumption to be the current bandwidth consumption associated with that flow; and

predicting, for an active flow having a current bandwidth consumption greater than the computed available bandwidth, the bandwidth consumption to be the computed available bandwidth.

21. The computing device of claim 20 , wherein the processing instructions further cause the processor to perform the steps of:

generating a plurality of file block placement options for a file block, each file block placement option being associated with at least one of a plurality of data storage nodes in the distributed network, the file block placement options being based on a set of network parameters associated with the distributed network; and

selecting one of the plurality of file block placement options based at least in part on the predicted bandwidth consumptions.

22. The computing device of claim 21 , wherein the processing instructions further cause the processor to perform the step of determining whether each of the plurality of file block placement options satisfies requirements associated with users of the distributed network.

23. The computing device of claim 22 , wherein the set of network parameters includes one or more constraints, and wherein the processing instructions further cause the processor to perform the step of, when none of the plurality of file block placement options satisfies the requirements associated with the users of the distributed network:

relaxing at least one of the constraints;

generating a second plurality of file block placement options for the file block using the relaxed at least one constraint;

predicting a bandwidth consumption for each active flow for a candidate file block placement option of the second plurality of file block placement options; and

selecting one of the second plurality of file block placement options based at least in part on the predicted bandwidth consumptions associated with the candidate file block placement option of the second plurality of file block placement options.

24. The computing device of claim 22 , wherein the processing instructions further cause the processor to perform the step of, when none of the plurality of file block placement options satisfies the requirements associated with the users of the distributed network:

reducing the requirements associated with users of the distributed network by excluding at least one of the requirements associated with at least one of the users of the distributed network; and

determining whether each of the plurality of file block placement options satisfies the reduced requirements.

25. The computing device of claim 22 , wherein the processing instructions further cause the processor to perform the step of, when none of the plurality of file block placement options satisfies the requirements associated with the users of the distributed network, delaying selecting one of the plurality of file block placement options.

26. The computing device of claim 22 , wherein the processing instructions further cause the processor to perform the step of, when none of the plurality of file block placement options satisfies the requirements associated with the users of the distributed network:

predicting a bandwidth consumption for at least some of the active flows for a candidate file block placement option of the plurality of file block placement options taking into account suspension of one or more existing block replication processes;

selecting one of the plurality of file block placement options based at least in part on the predicted bandwidth consumptions taking into account the suspension of one or more existing block replication processes; and

suspending the one or more existing block replication processes.

27. A non-transitory computer-readable medium storing computer-executable instructions that, when executed by a processor, perform a method for predicting available bandwidth for a file block placement of a candidate flow on a link in a distributed network, the method comprising:

obtaining information about a plurality of flows carried by the link, the information about the plurality of flows carried by the link including a current bandwidth consumption for each flow of the plurality of flows carried by the link;

identifying whether each flow of the plurality of flows has a local constraint or a remote constraint, wherein the local constraint is a throughput constraint of the each flow of the link and the remote constraint is a throughput constraint of each flow of another physical link other than the link; and

computing the available bandwidth for the candidate flow based at least in part on the information about a plurality of flows carried by the link and the identification of whether each flow of the plurality of flows has a local constraint or a remote constraint to generate a predicted available bandwidth for the candidate flow;

selecting one of a plurality of file block placement options based at least in part on the predicted available bandwidth which meets a particular quality of service (QoS) requirement;

determining a set of storage nodes which correspond to the one of the plurality of file block placement options as selected; and

performing the file block placement to the set of storage nodes, wherein computing the available bandwidth for the candidate flow comprises:

subtracting a sum of the current bandwidth consumption of the flows carried by the link having a remote constraint from a capacity of the link; and

dividing the difference by the number of flows carried by the link plus one minus the number of flows carried by the link having a remote constraint.