Load balancing Hadoop distributed file system operations in a non-native operating system
Implementations are provided herein for utilizing the two-part nature of HDFS protocol communications received in a non-native HDFS environment to use discriminative information learned in the NameNode request to make the client experience more efficient. NameNode requests can be received by any node among a cluster of nodes. It can be appreciated that in some non-native HDFS, clients can communicate with any node among a cluster of nodes to perform transactions and need not first communicate with a NameNode. In addition, any node among the cluster of nodes has access to any data within the HDFS. Thus, upon receiving a NameNode request, the non-native HDFS Cluster of Nodes can more effectively load balance the cluster by directing HDFS clients to connect with Nodes that are less busy than other nodes. In this sense, a more efficient client experience can be offered for HDFS clients within a non-native HDFS environment.
1. A method comprising:
receiving a first part Hadoop Distributed File System (“HDFS”) Protocol request by a first node among a cluster of nodes operating as a non-native HDFS, wherein all nodes among the cluster of nodes are capable of receiving the first part HDFS Protocol request, wherein the HDFS Protocol request includes a HDFS NameNode request, wherein the cluster of nodes are associated with an access zone, wherein the first node is configured to send the HDFS NameNode request to a predetermined node within the cluster of nodes configured to process the HDFS Protocol request, wherein the first part HDFS protocol request includes a request for a Data Node associated with a set of blocks associated with at least one file, and wherein all nodes within the cluster of nodes can serve the set of blocks;
in response to receiving the first part HDFS protocol request:
determining a set of available internet protocol (“IP”) addresses;
scoring each IP address in the set of available IP addresses based on a number of active TCP connections on a node associated with each IP address;
determining a subset of the set of available IP addresses based on a replication factor and the scoring; and
responding to the first part HDFS protocol request with the subset of the set of available IP addresses.
2. The method of claim 1 , wherein the set of available IP addresses are based on an access zone associated with the first part HDFS protocol request, a subnet associated with the first part HDFS protocol request, and a rack associated with the first part HDFS protocol request.
3. The method of claim 1 , further comprising:
receiving a blacklist of IP addresses within the first part HDFS protocol request, wherein the determining the subset of the set of available IP addresses is further based on the blacklist of IP addresses.
4. The method of claim 3 , wherein the determining the subset of the set of available IP addresses first removes the blacklist of IP addresses from the set of available IP addresses.
5. The method of claim 1 , wherein a number of IP addresses in the subset of the set of available IP addresses is the same as the replication factor.
6. The method of claim 1 , wherein the determining the subset of the set of available IP addresses is based on comparing the score of each IP address to an active TCP connection threshold.
7. A system comprising at least one node in a cluster of nodes operating as a non-native HDFS, wherein the at least one node comprises at least one storage device and at least one hardware processor configured to:
receive a first part Hadoop Distributed File System (“HDFS”) Protocol request by a first node among the cluster of nodes, wherein the at least one nodes are all capable of receiving the first part HDFS Protocol request, wherein the HDFS Protocol request includes a HDFS NameNode request, wherein the cluster of nodes are associated with an access zone, wherein the first node is configured to send the HDFS NameNode request to a predetermined node within the cluster of nodes configured to process the HDFS Protocol request, wherein the first part HDFS protocol request includes a request for a Data Node associated with a set of blocks associated with at least one file, and wherein the at least one nodes are all capable of serving the set of blocks;
in response to receiving the first part HDFS protocol request:
determine a set of available internet protocol (“IP”) addresses;
scoring each IP address in the set of available IP addresses based on a number of active TCP connections on a node associated with each IP address;
determine a subset of the set of available IP addresses based on a replication factor and the scoring; and
respond to the first part HDFS protocol request with the subset of the set of available IP addresses.
8. The system of claim 7 , wherein the set of available IP addresses are based on an access zone associated with the first part HDFS protocol request, a subnet associated with the first part HDFS protocol request, and a rack associated with the first part HDFS protocol request.
9. The system of claim 7 , further configured to:
receive a blacklist of IP addresses within the first part HDFS protocol request, wherein the determining the subset of the set of available IP addresses is further based on the blacklist of IP addresses.
10. The system of claim 9 , wherein the determining the subset of the set of available IP addresses first removes the blacklist of IP addresses from the set of available IP addresses.
11. The system of claim 7 , wherein a number of IP addresses in the subset of the set of available IP addresses is the same as the replication factor.
12. The system of claim 7 , wherein the determining the subset of the set of available IP addresses is based on comparing the score of each IP address to an active TCP connection threshold.
13. A non-transitory computer readable medium with program instructions stored thereon to perform the following acts:
receiving a first part Hadoop Distributed File System (“HDFS”) Protocol request by a first node among the cluster of nodes operating as a non-native HDFS, wherein all nodes among the cluster of nodes are capable of receiving the first part HDFS Protocol request, wherein the HDFS Protocol request includes a HDFS NameNode request, wherein the cluster of nodes are associated with an access zone, wherein the first node is configured to send the HDFS NameNode request to a predetermined node within the cluster of nodes configured to process the HDFS Protocol request, wherein the first part HDFS protocol request includes a request for a Data Node associated with a set of blocks associated with at least one file, and wherein all nodes within the cluster of nodes can serve the set of blocks;
in response to receiving the first part HDFS protocol request:
determining a set of available internet protocol (“IP”) addresses;
scoring each IP address in the set of available IP addresses based on a number of active TCP connections on a node associated with each IP address;
determining a subset of the set of available IP addresses based on a replication factor and the scoring; and
responding to the first part HDFS protocol request with the subset of the set of available IP addresses.
14. The non-transitory computer readable medium of claim 13 , wherein the set of available IP addresses are based on an access zone associated with the first part HDFS protocol request, a subnet associated with the first part HDFS protocol request, and a rack associated with the first part HDFS protocol request.
15. The non-transitory computer readable medium of claim 13 with program instructions stored thereon to further perform the following acts:
receiving a blacklist of IP addresses within the first part HDFS protocol request, wherein the determining the subset of the set of available IP addresses is further based on the blacklist of IP addresses.
16. The non-transitory computer readable medium of claim 15 , wherein the determining the subset of the set of available IP addresses first removes the blacklist of IP addresses from the set of available IP addresses.
17. The non-transitory computer readable medium of claim 13 , wherein a number of IP addresses in the subset of the set of available IP addresses is the same as the replication factor.
18. The non-transitory computer readable medium of claim 13 , wherein the determining the subset of the set of available IP addresses is based on comparing the score of each IP address to an active TCP connection threshold.