Integrating host-side storage device management with host-side non-volatile memory

1. A method for managing a data storage system including a set of flash-memory storage devices, each storage device including a write buffer and memory and supporting multi-stream data write, and a host coupled to the set of storage devices, wherein the write buffer of each storage device comprises volatile memory and is not protected against power loss, the method comprising:

allocating space in a memory of the host for a front-end write buffer;

storing, in the memory of the host, a first mapping table for data stored in the front-end write buffer and a second mapping table for data stored in the memory of each storage device;

managing, by the host, data storage functions for each storage device, the data storage functions including: address mapping, managing the write buffer of each storage device, scheduling reads and writes for the memory of each storage device, and controlling movement of data from the write buffer of each storage device to the memory of the storage device; and,

for each storage device, storing data from a block in the write buffer of the storage device to a parallel unit in the memory of the storage device if a write priority factor of the parallel unit is greater than a read priority factor of the parallel unit, wherein the read priority factor of the parallel unit is based on an amount of data read from the parallel unit over a fixed period of time, and wherein the write priority factor of the parallel unit is based on the amount of data in the front-end buffer that is associated with the parallel unit.

2. The method according to claim 1 , further comprising, in response to a read request from the host:

looking up a read address for the read request in the first mapping table;

if the read address is not found in the first mapping table, looking up the read address in the second mapping table to determine the storage device associated with the read address, fetching data from the memory of the storage device associated with the read address, and sending the data to the host; and

if the read address is found in the first mapping table, fetching data at the read address in the front-end write buffer and sending the data to the host.

3. The method according to claim 1 , further comprising, in response to a write request from the host:

looking up a write address for the write request in the first mapping table;

if the write address is not found in the first mapping table, writing at least a portion of data in the write request to the front-end write buffer;

if the write address is found in the first mapping table, writing the data to the write address in the front-end write buffer, determining if the data has already been copied from the front-end buffer to one of the storage devices, and if the data has already been copied to one of the storage devices, writing the data to the write buffer of that storage device.

4. A host-managed data storage system, comprising:

a set of storage devices, each storage device including a write buffer and memory, wherein the write buffer of each storage device comprises volatile memory and is not protected against power loss; and

a host coupled to the set of storage devices, the host including:

a storage device management module for managing data storage functions for each storage device, wherein the data storage functions for each storage device managed by the storage device management module include: address mapping, management of the write buffer of each storage device, and read and write scheduling for the memory of each storage device;

memory including: a front-end write buffer; a first mapping table for data stored in the front-end write buffer; and a second mapping table for data stored in the memory of each storage device,

wherein each storage device supports multi-stream data write, and wherein the write buffer of each storage device has a size of k·n c , where k is a constant and n c denotes the amount of data being written to the memory of the storage device in each stream at the same time,

wherein each storage device includes a number of parallel units and each parallel unit has a number of write streams, and wherein a total capacity of the write buffer of each storage device is independent from the number of parallel units in the storage device and the number of write streams in each parallel unit,

wherein, for each storage device, the storage device management module controls movement of data from the write buffer of the storage device to the memory of the storage device,

wherein, for each storage device, the storage device management module stores data from a block in the write buffer of the storage device to a parallel unit in the memory of the storage device if a write priority factor of the parallel unit is greater than a read priority factor of the parallel unit, wherein the read priority factor of the parallel unit is based on an amount of data read from the parallel unit over a fixed period of time, and wherein the write priority factor of the parallel unit is based on the amount of data in the front-end buffer that is associated with the parallel unit.

5. The storage system according to claim 4 , wherein each storage device comprises a NAND flash memory storage device.

6. The storage system according to claim 4 , wherein the set of storage devices support a total of d·p·m streams, where d denotes the number of storage devices in the set of storage devices, p denotes the number of parallel units supported by each storage device, and m denotes the number of streams supported in each parallel unit.

7. The storage system according to claim 6 , wherein the front-end buffer in the memory on the host has a capacity of d·p·m·n c .

8. The storage system according to claim 4 , wherein, in response to a read request from the host, the storage device management module is configured to:

look up a read address for the read request in the first mapping table; and

if the read address is not found in the first mapping table, look up the read address in the second mapping table to determine the storage device associated with the read address, fetch data from the memory of the storage device associated with the read address, and send the data to the host.

9. The storage system according to claim 4 , wherein, in response to a read request from the host, the storage device management module is configured to:

look up a read address for the read request in the first mapping table; and

if the read address is found in the first mapping table, fetch data at the read address in the front-end write buffer and send the data to the host.

10. The storage system according to claim 4 , wherein, in response to a write request from the host, the storage device management module is configured to:

look up a write address for the write request in the first mapping table; and

if the write address is not found in the first mapping table, write at least a portion of data in the write request to the front-end write buffer.

11. The storage system according to claim 4 , wherein, in response to a write request from the host, the storage device management module is configured to:

look up a write address for the write request in the first mapping table; and

if the write address is found in the first mapping table:

write the data to the write address in the front-end write buffer;

determine if the data has already been copied from the front-end buffer to one of the storage devices; and

if the data has already been copied to one of the storage devices, write the data to the write buffer of that storage device.

12. A host-managed data storage system, comprising:

a set of storage devices, each storage device including a write buffer and memory, wherein the write buffer of each storage device comprises volatile memory and is not protected against power loss; and

a host coupled to the set of storage devices, the host including:

a storage device management module for managing data storage functions for each storage device, wherein the data storage functions for each storage device managed by the storage device management module include: address mapping, management of the write buffer of each storage device, and read and write scheduling for the memory of each storage device;

memory including: a front-end write buffer; a first mapping table for data stored in the front-end write buffer; and a second mapping table for data stored in the memory of each storage device,

wherein each storage device supports multi-stream data write, and wherein the write buffer of each storage device has a size of k·n c , where k is a constant and n c denotes the amount of data being written to the memory of the storage device in each stream at the same time,

wherein the set of storage devices support a total of d·p·m streams, where d denotes the number of storage devices in the set of storage devices, p denotes the number of parallel units supported by each storage device, and m denotes the number of streams supported in each parallel unit,

wherein the front-end buffer in the memory on the host has a capacity of d·p·m·n c ,

wherein, for each storage device, the storage device management module controls movement of data from the write buffer of the storage device to the memory of the storage device,

wherein, for each storage device, the storage device management module stores data from a block in the write buffer of the storage device to a parallel unit in the memory of the storage device if a write priority factor of the parallel unit is greater than a read priority factor of the parallel unit, wherein the read priority factor of the parallel unit is based on an amount of data read from the parallel unit over a fixed period of time, and wherein the write priority factor of the parallel unit is based on the amount of data in the front-end buffer that is associated with the parallel unit.