Method for data transfer between compute clusters and file system
A data migrating system and method are provided in which a Burst Buffer Network Aggregator (BBNA) process is configured either on the File Servers or on the File System's dedicated I/O nodes to coalesce data fragments stored in participating Burst Buffer nodes under the direction of a primary BB node appointed by a data generating entity prior to transfer of the full data stripe into the File System. The “write” request in the form of a full data stripe is distributed into a plurality of data fragments among participating BB nodes along with corresponding metadata. The primary BB node gathers the metadata from the participating BB nodes, sends the metadata list to the BBNA unit, responsive to which the BBNA unit allocates a buffer sufficient to store the full data stripe, and transfers data fragments from participating BB nodes into the full data stripe buffer, thereby coalescing the data fragments into the full data stripe, which is subsequently transferred from the buffer in the BBNA unit into the File System.
1. A method for data migration between data generating entities and a file system, comprising the steps of:
(a) operatively coupling a Burst Buffer (BB) tier between at least one data generating entity and a File System, and configuring said BB tier with a first plurality of BB nodes, each operating with data input speed exceeding the data input speed of said File System;
(b) sending a first data READ request from said at least one data generating entity and assigning a primary BB node in said BB tier to service said first data READ request, said first data READ request including a plurality of data fragments related to a respective full data stripe;
(c) operatively coupling at least one Burst Buffer Network Aggregator (BBNA) processor unit to said File System, and configuring said at least one BBNA processor unit for allocation of a full data swipe buffer responsive to said first data READ request, and to issue a second data READ request subsequent to completion of the allocation of said full data stripe buffer;
(d) allocating, by said at least one BBNA processor unit, said full data stripe buffer sufficient to service said first data READ request, issuing, by said at least one BBNA processor unit, said second READ request, and sending said second READ request to said File System to read each of said plurality of data fragments of said full data stripe therefrom to said allocated full data stripe buffer;
(e) upon completion of said second READ request by said File System, issuing, by said at least one BBNA processor unit, a Remote Memory WRITE request; and
(f) selecting, by said primary BB node, a second plurality of destination BB nodes from said first plurality of BB nodes in said BB tier, and transferring each of said plurality of data fragments related to said full data stripe from said allocated full data stripe buffer to a respective destination BB node selected by said primary BB node from said second plurality of destination BB nodes, thus distributing said plurality of data fragments related to said full data stripe among said second plurality of said destination BB nodes selected by said primary BB node for servicing said first data READ request for being temporarily stored therein.
2. The method of claim 1 , further comprising the step of:
prior to said step (f), issuing, by said primary BB node, a READ command indicating said respective destination BB node for storing said each data fragment.
3. The method of claim 2 , further comprising the step of:
upon completion of said each data fragment transfer to said respective destination BB node in said step (f), reporting, by said at least one BBNA unit, to said primary BB node on a status of said each data fragment transfer.