Method and system for enhancing IOPS of a hard disk drive system based on storing metadata in host volatile memory and data in non-volatile memory using a shared controller

1. A computer-implemented method, comprising:

receiving, by a storage engine, a request to write data to a hard disk drive;

determining metadata associated with the data;

storing the metadata in a volatile memory associated with the storage engine, wherein the metadata is stored in the volatile memory of the storage engine as a first entry based on a metadata index and a plurality of metadata content chunks;

identifying a physical address in a first non-volatile solid-state memory to which to write the metadata, wherein the first non-volatile solid-state memory is accessible via a controller of the hard disk drive;

writing the metadata to the first non-volatile solid-state memory sequentially based on the identified physical address at a next available location of the first non-volatile solid-state memory,

wherein the metadata is written as a second entry corresponding to the first entry and wherein the second entry includes:

a Not-AND (NAND) offset index which corresponds to the metadata index;

a physical address for a corresponding metadata content chunk prior to a current update; and

the corresponding metadata content chunk; and

writing the data to the hard disk drive.

2. The method of claim 1 , further comprising:

determining that the metadata is an update to existing metadata stored in the storage engine; and

writing the updated metadata to the first non-volatile solid-state memory based on the identified physical address by appending the updated metadata at a next available location of the first non-volatile solid-state memory.

3. The method of claim 1 ,

wherein a second non-volatile solid-state memory is accessible via the controller of the hard disk drive,

wherein the first non-volatile solid-state memory is a first NAND die and the second non-volatile solid-state memory is a second NAND die, and

wherein a NAND die includes at least two planes.

4. The method of claim 3 , further comprising:

determining that the first NAND die stores an amount of data which is greater than a first predetermined threshold;

flushing metadata stored in the volatile memory of the storage engine to the second NAND die;

writing updated metadata to the volatile memory of the storage engine;

synchronizing the updated metadata by writing the updated metadata to the second NAND die; and

erasing the data from the first NAND die.

5. The method of claim 1 , wherein the method further comprises:

detecting an error associated with accessing the metadata stored in the volatile memory of the storage engine; and

reconstructing the metadata by:

scanning, based on the metadata index, the first non-volatile solid-state memory to obtain the second entry and any other entries which include the metadata index; and

selecting, from the second entry and the other entries, based on the prior physical address of the second entry or a respective prior physical address of the other entries, a third entry which corresponds to a most recent version of the metadata,

wherein the reconstructed metadata comprises a corresponding metadata content chunk included in the third entry.

6. The method of claim 1 , wherein the hard disk drive controller comprises one or more of:

a cyclic redundancy check (CRC) encoder or decoder module;

a run limited length/maximum transition run (RLL/MTR) encoder or decoder module;

an error correction code (ECC) encoder or decoder module;

a low-density parity-check (LDPC) encoder or decoder module;

an interleaver or de-interleaver module;

a log likelihood ratio (LLR) generator module;

a redundant array of interdependent disks (RAID) management module;

an interface to a write head or a read head; and

a Toggle/Open NAND Flash Interface (ONFI).

7. The method of claim 1 , further comprising:

receiving a request to read the data from the hard disk drive;

in response to successfully obtaining the associated metadata from the volatile memory of the storage engine, obtaining the requested data from the hard disk drive based on the associated metadata; and

in response to unsuccessfully obtaining the associated metadata from the volatile memory of the storage engine, obtaining the associated metadata from the first non-volatile solid-state memory.

8. The method of claim 7 , wherein obtaining the associated metadata from the first non-volatile solid-state memory involves operations by one or more modules of the hard disk drive controller, including:

a Toggle/Open NAND Flash Interface (ONFI);

a log likelihood ratio (LLR) generator module;

an error correction code (ECC) encoder or decoder module;

a low-density parity-check (LDPC) encoder or decoder module;

a redundant array of interdependent disks (RAID) management module; and

a cyclic redundancy check (CRC) encoder or decoder module.

9. A computer system, comprising:

a storage engine;

a hard disk drive, wherein a first non-volatile solid state memory is accessible via a controller of the hard disk drive, and

wherein the storage engine comprises a processor and a memory coupled to the processor and storing instructions which, when executed by the processor cause the processor to perform a method, the method comprising:

receiving a request to write data to a hard disk drive;

determining metadata associated with the data;

storing the metadata in a volatile memory associated with the storage engine, wherein the metadata is stored in the volatile memory of the storage engine as a first entry based on a metadata index and a plurality of metadata content chunks;

identifying a physical address in the first non-volatile solid-state memory to which to write the metadata;

writing the metadata to the first non-volatile solid-state memory sequentially based on the identified physical address at a next available location of the first non-volatile solid-state memory,

wherein the metadata is written as a second entry corresponding to the first entry and wherein the second entry includes:

a Not-AND (NAND) offset index which corresponds to the metadata index:

a physical address for a corresponding metadata content chunk prior to a current update; and

the corresponding metadata content chunk; and

writing the data to the hard disk drive.

10. The computer system of claim 9 , wherein the method further comprises:

determining that the metadata is an update to existing metadata stored in the storage engine; and

writing the updated metadata to the first non-volatile solid-state memory based on the identified physical address by appending the updated metadata at a next available location of the first non-volatile solid-state memory.

11. The computer system of claim 9 ,

wherein a second non-volatile solid-state memory is accessible via the controller of the hard disk drive,

wherein the first non-volatile solid-state memory is a first NAND die and the second non-volatile solid-state memory is a second NAND die, and

wherein a NAND die includes at least two planes.

12. The computer system of claim 11 , wherein the method further comprises:

determining that the first NAND die stores an amount of data which is greater than a first predetermined threshold;

flushing metadata stored in the volatile memory of the storage engine to the second NAND die;

writing updated metadata to the volatile memory of the storage engine;

synchronizing the updated metadata by writing the updated metadata to the second NAND die; and

erasing the data from the first NAND die.

13. The computer system of claim 9 , wherein the method further comprises:

detecting an error associated with accessing the metadata stored in the volatile memory of the storage engine; and

reconstructing the metadata by:

scanning, based on the metadata index, the first non-volatile solid-state memory to obtain the second entry and any other entries which include the metadata index;

selecting, from the second entry and the other entries, based on the prior physical address of the second entry or a respective prior physical address of the other entries, a third entry which corresponds to a most recent version of the metadata,

wherein the reconstructed metadata comprises a corresponding metadata content chunk included in the third entry.

14. The computer system of claim 9 , wherein the hard disk drive controller comprises one or more of:

a cyclic redundancy check (CRC) encoder or decoder module;

a run limited length/maximum transition run (RLL/MTR) encoder or decoder module;

an error correction code (ECC) encoder or decoder module;

a low-density parity-check (LDPC) encoder or decoder module;

an interleaver or de-interleaver module;

a log likelihood ratio (LLR) generator module;

a redundant array of interdependent disks (RAID) management module;

an interface to a write head or a read head; and

a Toggle/Open NAND Flash Interface (ONFI).

15. The computer system of claim 9 , wherein the method further comprises:

receiving a request to read the data from the hard disk drive;

in response to successfully obtaining the associated metadata from the volatile memory of the storage engine, obtaining the requested data from the hard disk drive based on the associated metadata; and

in response to unsuccessfully obtaining the associated metadata from the volatile memory of the storage engine, obtaining the associated metadata from the first non-volatile solid-state memory.

16. An apparatus, comprising:

a hard disk drive, wherein a first non-volatile solid state memory is accessible via a controller of the hard disk drive, and

a storage engine configured to:

receive a request to write data to a hard disk drive;

determine metadata associated with the data;

store the metadata in a volatile memory associated with the storage engine, wherein the metadata is stored in the volatile memory of the storage engine as a first entry based on a metadata index and a plurality of metadata content chunks;

identify a physical address in the first non-volatile solid-state memory to which to write the metadata;

write the metadata to the first non-volatile solid-state memory sequentially based on the identified physical address at a next available location of the first non-volatile solid-state memory,

wherein the metadata is written as a second entry corresponding to the first entry and wherein the second entry includes:

a Not-AND (NAND) offset index which corresponds to the metadata index:

a physical address for a corresponding metadata content chunk prior to a current update; and

the corresponding metadata content chunk; and

write the data to the hard disk drive.

17. The apparatus of claim 16 , wherein the storage engine is further configured to:

determine that the metadata is an update to existing metadata stored in the storage engine; and

write the updated metadata to the first non-volatile solid-state memory based on the identified physical address by appending the updated metadata at a next available location of the first non-volatile solid-state memory.

18. The apparatus of claim 16 ,

wherein a second non-volatile solid-state memory is accessible via the controller of the hard disk drive,

wherein the first non-volatile solid-state memory is a first NAND die and the second non-volatile solid-state memory is a second NAND die, and

wherein a NAND die includes at least two planes.

19. The apparatus of claim 18 , wherein the controller of the hard disk drive is configured to:

determine that the first NAND die stores an amount of data which is greater than a first predetermined threshold;

flush metadata stored in the volatile memory of the storage engine to the second NAND die,

wherein the storage engine is further configured to write updated metadata to the volatile memory of the storage engine;

synchronize the updated metadata by writing the updated metadata to the second NAND die; and

erase the data from the first NAND die.

20. The apparatus of claim 16 , wherein the storage engine is further configured to:

detect an error associated with accessing the metadata stored in the volatile memory of the storage engine; and

reconstruct the metadata by:

scanning, based on the metadata index, the first non-volatile solid-state memory to obtain the second entry and any other entries which include the metadata index; and

selecting, from the second entry and the other entries, based on the prior physical address of the second entry or a respective prior physical address of the other entries, a third entry which corresponds to a most recent version of the metadata,

wherein the reconstructed metadata comprises a corresponding metadata content chunk included in the third entry.