Distributed application framework that uses network and application awareness for placing data
A distributed application framework, along with related systems and/or methods described herein, can intelligently place data using network knowledge. An exemplary method can include receiving data placement information from a distributed application that identifies a source node of data in a network and a list of potential destination nodes in the network for the distributed application to place the data; for each potential destination node, determining a network latency associated with transferring the data from the source node to the potential destination node using network metrics associated with the network; and sending the determined network latencies to the distributed application, such that the distributed application can assign the data to one of the potential destination nodes based on the determined network latencies.
1. A method, comprising:
receiving data placement information from a distributed application that identifies a source node of data in a network and a list of potential destination nodes in the network for the distributed application to place the data;
for each potential destination node, determining a network latency associated with transferring the data from the source node to the potential destination node using network metrics associated with the network; and
sending the determined network latencies to the distributed application, such that the distributed application can assign the data to one of the potential destination nodes based on the determined network latencies;
wherein the determining network latency comprises determining an overall latency (OL) from the source node (SN) to the potential destination node (PDN), as follows:
OL=K×[A×(Number of Hops Between SN and PDN)]×[B×(Aggregate Latency of Network Elements associated with Hops Between SN and PDN)]×[C×(Aggregate Time for Transferring Data based on Data Size and Network Bandwidth of Each Network Element associated with Hops Between SN and PDN]×[D×(Aggregate Time for Transferring Data based on Data Size and Planned Available Bandwidth Metric of Each Network Element associated with Hops Between SN and PDN)]×[E×(an estimate of input/output bandwidth for PDN for a defined time in the future)]×[F×(current input/output operations per second PDN)];
where A, B, C, D, E, F and K are non-zero constants.
2. The method of claim 1 , further comprising, based on the determined network latencies, identifying a best potential destination node from the list of potential destination nodes for placing the data.
3. The method of claim 2 , wherein the best potential destination node exhibits lowest network latency.
4. The method of claim 2 , further comprising ranking potential destination nodes having a same determined network latency by weighting one network metric over another network metric.
5. The method of claim 1 , further comprising generating planned network metrics for the network, wherein the planned network metrics indicate future utilization of the network, and further wherein the network metrics used to determine the network latency include the planned network metrics.
6. The method of claim 5 , further comprising updating the planned network metrics once the distributed application assigns the data to one of the potential destination nodes.
7. The method of claim 5 , wherein generating the planned network metrics includes determining storage capabilities for each of the potential destination nodes.
8. The method of claim 1 , further comprising collecting the network metrics from the network and the distributed application.
9. The method of claim 1 , wherein for each potential destination node, the network latency is a function of one or more of the following network metrics:
a number of hops between the source node and the potential destination node;
an aggregate network latency of network elements associated with each hop between the source node and the potential destination node;
an aggregate time for transferring data based on data size and network bandwidth associated with each hop between the source node and the potential destination node;
an aggregate time for transferring data based on data size and planned available network bandwidth associated with each hop between the source node and the potential destination node;
a planned input/output bandwidth for the potential destination node for the future time period; and
an input/output operations per second for the potential destination node.
10. The method of claim 9 , further comprising weighting the network metrics to determine the network latency.
11. Logic encoded in non-transitory media that includes instructions for execution and when executed by a processor, is operable to perform operations comprising: receiving data placement information from a distributed application that identifies a source node of data in a network and a list of potential destination nodes in the network for the distributed application to place the data;
for each potential destination node, determining a network latency associated with transferring the data from the source node to the potential destination node using network metrics associated with the network; and
sending the determined network latencies to the distributed application, such that the distributed application can assign the data to one of the potential destination nodes based on the determined network latencies;
wherein the determining network latency comprises determining an overall latency (OL) from the source node (SN) to the potential destination node (PDN), as follows:
OL=K×[A×(Number of Hops Between SN and PDN)]×[B×(Aggregate Latency of Network Elements associated with Hops Between SN and PDN)]×[C×(Aggregate Time for Transferring Data based on Data Size and Network Bandwidth of Each Network Element associated with Hops Between SN and PDN]×[D×(Aggregate Time for Transferring Data based on Data Size and Planned Available Bandwidth Metric of Each Network Element associated with Hops Between SN and PDN)]×[E×(an estimate of input/output bandwidth for PDN for a defined time in the future)]×[F×(current input/output operations per second PDN)];
where A, B, C, D, E, F and K are non-zero constants.
12. The logic of claim 11 , the operations further including, based on the determined network latencies, identifying a best potential destination node from the list of potential destination nodes for placing the data.
13. The logic of claim 12 , the operations further including ranking potential destination nodes having a same determined network latency by weighting one network metric over another network metric.
14. The logic of claim 13 , the operations further including, generating planned network metrics for the network, wherein the planned network metrics indicate future utilization of the network, and further wherein the network metrics used to determine the network latency include the planned network metrics.
15. The logic of claim 14 , the operations further including updating the planned network metrics once the distributed application assigns the data to one of the potential destination nodes.
16. The logic of claim 11 , wherein for each potential destination node, the network latency is a function of one or more of the following network metrics:
a number of hops between the source node and the potential destination node;
an aggregate network latency of network elements associated with each hop between the source node and the potential destination node;
an aggregate time for transferring data based on data size and network bandwidth associated with each hop between the source node and the potential destination node;
an aggregate time for transferring data based on data size and planned available network bandwidth associated with each hop between the source node and the potential destination node;
a planned input/output bandwidth for the potential destination node for the future time period; and
an input/output operations per second for the potential destination node.
17. An apparatus comprising:
a memory element for storing data; and
a processor operable to execute instructions associated with the data, wherein the processor and the memory element cooperate such that the apparatus is configured for:
receiving data placement information from a distributed application that identifies a source node of data in a network and a list of potential destination nodes in the network for the distributed application to place the data;
for each potential destination node, determining a network latency associated with transferring the data from the source node to the potential destination node using network metrics associated with the network; and
sending the determined network latencies to the distributed application, such that the distributed application can assign the data to one of the potential destination nodes based on the determined network latencies;
wherein the determining network latency comprises determining an overall latency (OL) from the source node (SN) to the potential destination node (PDN), as follows:
OL=K×[A×(Number of Hops Between SN and PDN)]×[B×(Aggregate Latency of Network Elements associated with Hops Between SN and PDN)]×[C×(Aggregate Time for Transferring Data based on Data Size and Network Bandwidth of Each Network Element associated with Hops Between SN and PDN]×[D×(Aggregate Time for Transferring Data based on Data Size and Planned Available Bandwidth Metric of Each Network Element associated with Hops Between SN and PDN)]×[E×(an estimate of input/output bandwidth for PDN for a defined time in the future)]×[F×(current input/output operations per second PDN)];
where A, B, C, D, E, F and K are non-zero constants.
18. The apparatus of claim 17 , further configured for, based on the determined network latencies, identifying a best potential destination node from the list of potential destination nodes for placing the data.
19. The apparatus of claim 18 , further configured for ranking potential destination nodes having a same determined network latency by weighting one network metric over another network metric.
20. The apparatus of claim 17 , further configured for generating planned network metrics for the network, wherein the planned network metrics indicate future utilization of the network, and further wherein the network metrics used to determine the network latency include the planned network metrics.