Method and system for data transfer between compute clusters and file system

1. A method for data migration between data generating entities and a File System, comprising:

(a) operatively coupling a Burst Buffer (BB) tier between at least one data generating entity and a File System, and configuring said BB tier with a plurality of Burst Buffer (BB) nodes, each operating with data input speed exceeding the data input speed of said File System;

(b) configuring said File System with at least one Burst Buffer Network Aggregator (BBNA) unit;

(c) sending a first data write request from said at least one data generating entity to said BB tier, said first data write request including a plurality of data fragments related to a respective data stripe;

(d) distributing said first data write request into said plurality of data fragments, and storing each of said plurality of data fragments and metadata associated therewith in a respective participating BB node from said plurality of BB nodes in said BB tier;

(e) assigning a primary BB node from said participating BB nodes for servicing said first data write request;

(f) gathering, by said primary BB node, metadata for said plurality of data fragments residing in said participating BB nodes, and generating a metadata list corresponding to said plurality of data fragments for said respective data stripe;

(g) sending a command to coalesce said plurality of data fragments for said respective data stripe along with said metadata list from said primary BB node to said at least one BBNA unit;

(h) subsequently, allocating, by said at least one BBNA unit, a full data stripe buffer sufficient to accommodate said first data write request;

(i) transferring said plurality of data fragments from said participating BB nodes into said full data stripe buffer in accordance with said metadata list, thereby coalescing said plurality of data fragments of said data write request into a full said respective data stripe in said full data stripe buffer; and

(j) transferring said full data stripe from said full data stripe buffer of said at least one BBNA unit into said File System.

2. The method of claim 1 , further comprising:

prior to said step (j), issuing a second data write request by said at least one BBNA unit to said File System.

3. The method of claim 1 , further comprising:

upon completion of said data transfer in said step (j), reporting, by said at least one BBNA unit to said primary BB node.

4. The method of claim 1 , wherein each of said BB nodes allocates a byte addressable buffer, further comprising:

servicing said first data write request in a BBNA Target Mode, and

prior to said step (i), issuing a remote memory READ request by said at least one BBNA unit to read contents of said byte addressable buffer of each said BB node.

5. The method of claim 4 , further comprising:

prior to said step (i), signaling, by said primary BB node to said plurality of participating BB nodes, to move data fragments residing thereat to said byte addressable buffers.

6. The method of claim 5 , further comprising:

in said step (f), gathering scattered information about said byte addressable buffers by said primary BB node.

7. A method for data migration between data generating entities a File System, comprising:

(a) operatively coupling a Burst Buffer (BB) tier between at least one data generating entity and a File System, and configuring said BB tier with a plurality of Burst Buffer (BB) nodes, each BB node being configured with a byte addressable buffer allocated therein, and each operating with data input speed exceeding the data input speed of said File System;

(b) configuring said File System with at least one Burst Buffer Network Aggregator (BBNA) unit;

(c) sending a first data write request from said at least one data generating entity to said BB tier;

(d) distributing said data into data fragments, and storing each data fragment and metadata associated therewith in a respective participating BB node from said plurality of BB nodes in said BB tier;

(e) assigning a primary BB node from said participating BB nodes for servicing said first data write request;

(f) gathering, by said primary BB node, metadata for said data fragments residing in said participating BB nodes, and generating a metadata list;

(g) sending a command along with said metadata list from said primary BB node to said at least one BBNA unit;

(h) subsequently, allocating, by said at least one BBNA unit, a full data stripe buffer sufficient to accommodate said first data write request;

(i) transferring said data fragments from said participating BB nodes into said full data stripe buffer in accordance with said metadata list, thereby coalescing said data fragments of said data write segment into a full data stripe in said fill data stripe buffer;

(j) creating by said primary BB node, a command including BB nodes ID, buffer descriptor, address, size, and a RDMA (Remote Direct Memory Access) key on said byte addressable buffers, and

(k) transferring said full data stripe from said full data stripe buffer of said at least one BBNA unit into said File System.

8. The method of claim 1 , wherein said File System includes a plurality of File Servers, further comprising:

in said step (b), configuring said at least one BBNA unit on at least one of said plurality of File Servers.

9. The method of claim 1 , further comprising:

operatively coupling at least one File System Input/Output node between said at least one data generating entity and said File System,

dedicating said at least one File System Input/Output node for interaction with said File System, and

in said step (b), configuring said at least one BBNA unit on said at least one File System Input/Output node.

10. The method of claim 1 , further comprising:

servicing said first data write request in a BBNA Host Mode,

prior to said step (i), issuing a READ request by said at least one BBNA unit to said participating BB nodes,

responsive to said READ request, pushing data fragments residing at said participating BB nodes to said full data stripe buffer allocated by said at least one BBNA unit.

11. A system for data migration between data generating entities and a File System, comprising:

a Burst Buffer (BB) tier operatively coupled between at least one data generating and the File System, wherein said Burst Buffer tier includes a plurality of BB nodes having data input speed exceeding the data input speed of said File System;

a plurality of File Servers operatively coupled to said BB tier; and

at least one Burst Buffer Network Aggregator (BBNA) unit configured at said File System in operative coupling to said plurality of BB nodes, wherein each of said BB nodes is configured with a byte addressable buffer, wherein said at least one BBNA unit is configured to read contents of said byte addressable buffer, and wherein said at least one BB node is configured to create a command including BB nodes ID and burst buffer descriptor, size, and a RDMA (Remote Direct Memory Access) key on said byte addressable buffer,

said at least one BBNA unit being configured to coalesce data fragments stored in participating BB nodes from said plurality thereof into a full data stripe under the direction of at least one of said participating BB nodes prior to transfer said full data stripe into said File System.

12. The system of claim 11 , wherein said at least one BBNA unit resides on a respective one of said plurality of File Servers.

13. The system of claim 11 , wherein said File System further comprises a plurality of File System Input/Output (FSIO) nodes operatively coupled to said plurality of data generating entities and dedicated for said File System interaction, wherein said at least one BBNA unit resides on a respective one of said plurality of FSIO nodes.

14. The system of claim 11 , wherein each of said plurality of BB nodes is configured to store at least one data fragment of said full data stripe and metadata associated therewith,

wherein said at least one BB node is configured to gather said metadata from said plurality of participating BB nodes, and to generate a metadata list on data fragments residing in said participating BB nodes, and

wherein said at least one BBNA unit is configured to allocate a full data stripe buffer sufficient to store said full data stripe formed of said data fragments upon receipt of said metadata list from said at least one BB node, and to subsequently transfer said data fragments from said participating BB nodes to said full data stripe buffer prior to transferring said full data stripe to said File System.

15. The system of claim 14 , further comprising an array of data storage units operatively coupled to said plurality of File Servers.

16. The system of claim 11 , wherein, for servicing a READ request from at least one data generating entity, said at least one BBNA unit is configured to:

(a) allocate a buffer sufficient to accommodate said READ request;

(b) transfer said requested data from said File System to said buffer; and

(c) subsequently transfer said data from said buffer to at least one destination BB node indicated by said at least one BB node.

17. The system of claim 11 , further comprising a High Speed Network intercoupled between said data generating entities and said BB tier.

18. The system of claim 11 , further comprising a Storage Fabric intercoupled between said BB tier and said File System.

19. The system of claim 11 , further comprising a plurality of I/O nodes between said data generating entities and said File System, wherein each of said plurality of I/O nodes is augmented with a respective Burst Buffer unit.