Centralized parallel burst engine for high performance computing
Data storage systems and methods for storing data are described herein. An intermediate storage system is coupled with a High Performance Computing (HPC) system. A plurality of burst engines of the intermediate storage system are directly connected to compute nodes of the HPC. The burst engines are operative to receive file level data from the compute nodes at data transfer rates that at least match to the burst mode output rate of the compute node. The burst engines store the data to high bandwidth storage devices of the intermediate storage system at a data rate that at least matches the burst output rate of the compute node. During a compute phase of the compute node, data is replayed from the intermediate storage devices to a primary storage system at a lower data transfer rate.
1. A method for transferring data from at least a first compute node of a high performance computing (HPC) system, the method comprising:
capturing file level data output during a burst output phase at a software client supported at a compute node of a HPC system;
transferring said file level data and associate metadata, via an I/O node of said compute node, at a first data transfer rate to a temporary storage system;
receiving, from said compute node, file level data and associate metadata, via a front-end interface of a burst engine of said temporary storage system, at said first data transfer rate;
caching said file level data and associated metadata at an internal memory of said temporary storage system;
writing said file level data and associated metadata as block level data, via a back-end interface of said burst engine, to a set of dedicated non-volatile storage devices of said temporary storage system, where an aggregate data transfer rate of said set of storage devices equals or exceeds said first data transfer rate;
receiving a notification at a server, supported at said temporary storage system, from said software client at the compute node of the HPC system, said notification including logical block addresses of the block level data stored in the dedicated non-volatile storage devices of the temporary storage system;
retrieving block level data from said storage devices at said server;
interpreting said metadata at said server; and
replaying said block level data from said temporary storage system to a second storage system, in accordance with said metadata, at a second data transfer rate that is less than said first data transfer rate, wherein said block level data is replayed during a compute phase of said compute node of said HPC system.
2. The method of claim 1 , wherein replaying said block level data further comprises said server negotiating with said compute node to identify block level storage locations of said file level data.
3. The method of claim 1 , wherein writing said file level data as block level data further comprises:
writing said block level data with redundancy to said storage devices.
4. The method of claim 3 , wherein writing said block level data with redundancy further comprises:
writing said block level data with RAID redundancy.
5. The method of claim 3 , further comprising:
performing a XOR operation prior to writing said block level data to said storage devices.
6. The method of claim 1 , further comprising:
said server notifying said compute node that said storage devices are available upon completing replay of said block level data to said second storage system.
7. The method of claim 1 , wherein capturing file level data further comprises:
capturing data at file level at said compute node using a burst buffer client.