Systems and methods for routing data to a parallel file system
Systems and methods for routing data from a node to a parallel file system are disclosed. In some embodiments, a network system can include nodes, parallel file systems, segments, a control server, an endpoint device, and an access point server. Each of the segments can connect two nodes. The access point server and the endpoint device can be connected with a first tunnel. The access point server and the control server can be connected with a second tunnel.
1. A network system for routing data from a node to a parallel file system, comprising:
a first network node communicatively coupled to:
a plurality of parallel file systems;
a plurality of segments, each segment connecting two nodes;
a control server;
an endpoint device;
an access point server;
a first tunnel connecting the access point server to the endpoint device; and
a second tunnel connecting the access point server to the control server;
wherein the first network node is configured to send the data to a first parallel file system of the plurality of parallel file systems, and wherein sending the data comprises uniquely identifying a folder group among a plurality of folder groups of the first parallel file system to write the data to,
wherein each of the plurality of folder groups comprises:
a first folder assigned to the first network node, wherein the first network node batch processes files received in the first folder at a first predetermined time interval;
a second folder assigned to the first network node, wherein the first network node batch processes files received in the second folder at a second predetermined time interval that is longer than the first predetermined time interval, and
wherein each of the plurality of folder groups being assigned to a different network node among a plurality of network nodes that read the data from the first parallel file system.
2. The network system of claim 1 , wherein the first parallel file system is selected based on stored state information.
3. The network system of claim 2 , wherein the stored state information includes information of the plurality of parallel file systems on availability, location, and folders.
4. The network system of claim 1 , wherein sending the data to the first parallel file system includes directly writing the data to the first parallel file system or load-balanced writing the data to the first parallel file system.
5. The network system of claim 1 , wherein the first network node is located in a first region and the first parallel file system is located in a second region.
6. The network system of claim 5 , wherein the first region and the second region are different.
7. The network system of claim 1 , wherein each of the plurality of folder groups is associated with a quality of service (QoS).