System and method to improve enterprise reliability through tracking I/O performance metrics in non-volatile random access memory
View Patent ↗A method for managing a non-volatile random-access memory (NVRAM)-based storage subsystem, the method including: monitoring, by a slave controller on a NVRAM device of the NVRAM-based storage subsystem, an I/O operation on the NVRAM device; identifying, by the slave controller and based on the monitoring, at least one occurrence of error data; comparing, by the slave controller, a numeric aspect of the at least one occurrence of error data with a threshold setting; reporting, by the slave controller on the NVRAM device and to a master controller of the NVRAM-based storage subsystem, the at least one occurrence of error data in response to the numeric aspect crossing the threshold setting; and ascertaining, by the master controller of the NVRAM-based storage system, a physical location of a defect region on the NVRAM device where the error data has occurred by analyzing the reported at least one occurrence of error data.
1. A method for managing a non-volatile random-access memory (NVRAM)-based storage subsystem, the method comprising:
monitoring, by a slave controller on a NVRAM device of the NVRAM-based storage subsystem, input/output (I/O) operations on the NVRAM device;
measuring, by the slave controller on the NVRAM device, a duration of completing a particular I/O operation on the NVRAM device, and then comparing measured duration of the particular I/O operation to a time-out value indicating an upper limit for the I/O operation on the NVRAM device beyond which the I/O operation on the NVRAM will be aborted;
responsive to determining that the measured duration of the particular I/O operation exceeds the time-out value, measuring, by the slave controller on the NVRAM device, a performance characteristic associated with I/O operation on the NVRAM device, wherein the performance characteristic is different from the duration of the particular I/O operation;
comparing, by the slave controller on the NVRAM device, the measured performance characteristic to a threshold setting;
reporting, by the slave controller on the NVRAM device and to a master controller of the NVRAM-based storage subsystem, information based on results of the comparison of the measured performance characteristic to the threshold setting; and
determining, by the master controller of the NVRAM-based storage system and based on reported information, a physical location of a defect region on the NVRAM device where the performance characteristic was measured.
2. The method of claim 1 , further comprising:
summarizing information encoding physical locations of defect regions on multiple NVRAM devices of the NVRAM-based storage subsystem; and
storing the summarized information for persistent storage.
3. The method of claim 2 , further comprising:
driving decisions managing subsequent I/O operations on the multiple NVRAM devices of the NVRAM-based storage subsystem in view of the summarized information stored.
4. The method of claim 1 , wherein monitoring the I/O operations on the NVRAM device comprises monitoring at least one of: a read operation on the NVRAM device, a write operation on the NVRAM device, and an erase operation on the NVRAM device.
5. The method of claim 1 , wherein monitoring is performed more frequently during a first life-time phase of the NVRAM device than during a second life-time phase of the NVRAM device, wherein the first life-time phase is earlier than the second life-time phase.
6. The method of claim 1 , wherein measuring the performance characteristic comprises measuring a duration of completing a particular I/O operation on the NVRAM device.
7. The method of claim 1 , wherein measuring the performance characteristic comprises measuring a number of occurrences of correctable errors associated with the I/O operations on the NVRAM device, wherein the correctable error is curable by an error-checking mechanism.
8. The method of claim 1 , wherein measuring the performance characteristic further comprises measuring an error rate associated with the I/O operations on the NVRAM device.
9. The method of claim 1 , wherein measuring the performance characteristic of the NVRAM device further comprises measuring a number of occurrences of correctable errors associated with the I/O operations on the NVRAM device and wherein comparing the measured performance characteristic to the threshold setting comprises comparing the measured number of occurrences of correctable errors to the threshold setting.
10. The method of claim 1 , wherein measuring a performance characteristic of the NVRAM device comprises measuring an error rate associated with the I/O operations on the NVRAM device and wherein comparing the measured performance characteristic to the threshold setting comprises the measured error rate to the threshold setting.
11. The method of claim 1 , wherein the threshold setting relates to a statistical outlier, and wherein comparing the measured performance characteristic to the threshold setting comprises determining whether the measured performance characteristic qualifies as a statistical outlier.
12. The method of claim 11 , wherein determining whether the measured performance characteristic qualifies as a statistical outlier comprises determining whether the measured performance characteristic is more than a pre-determined number of standard deviations away from a statistical mean of past instances of the measured performance characteristic.
13. The method of claim 1 , further comprising: configuring the threshold setting.
14. The method of claim 1 , wherein ascertaining the physical location of the defect region comprises ascertaining whether the defect region corresponds to one of: a degraded region, or an inoperative region.
15. A non-volatile random access memory (NVRAM)-based storage subsystem mounted on a host computing device, the NVRAM-based storage subsystem comprising:
a plurality of NVRAM devices, each comprising a plurality of packages and a slave controller, and each being configured to:
monitor (input/output) I/O operations on the NVRAM device;
measure a duration of completing a particular I/O operation on the NVRAM device and then compare the measured duration of the particular I/O operation to a time-out value indicating an upper limit for the I/O operation on the NVRAM device beyond which the I/O operation on the NVRAM will be aborted;
responsive to determining that the measured duration of the particular I/O operation exceeding the time-out value, measure, by the slave controller on the NVRAM device, a performance characteristic associated with I/O operation on the NVRAM device, wherein the performance characteristic is different from the duration of the particular I/O operation;
compare the measured performance characteristic to a threshold setting; and
report information based on results of the comparison of the measured performance characteristic to the threshold setting;
a master controller, in communication with at least one of the slave controllers and configured to determine a physical location of a defect region on the NVRAM device where the performance characteristic was measured,
wherein each package comprises multiple dice, and
wherein each die comprises multiple planes.
16. The NVRAM-based storage subsystem of claim 15 , wherein each plane comprises at least one of: a Single-Level Cell (SLC) NAND flash, a Multi-Level Cell (MLC) NAND flash, a Single-Level Cell (SLC) NOR flash.
17. The NVRAM-based storage subsystem of claim 15 , wherein the I/O operation on the NVRAM device comprises one of: a read operation on the NVRAM device, a write operation on the NVRAM device, and an erase operation on the NVRAM device.
18. The NVRAM-based storage subsystem of claim 15 , wherein each slave controller is further configured to monitor the I/O operations more frequently during a first life-time phase of the NVRAM device than during a second life-time phase of the NVRAM device, wherein the first life-time phase is earlier than the second life-time phase.
19. The NVRAM-based storage subsystem of claim 15 , wherein each slave controller is further configured to measure: a number of occurrences of a correctable error associated with the I/O operations on the NVRAM device, or an error rate associated with the I/O operations on the NVRAM device, and wherein the correctable error is curable by an error-checking mechanism.
20. The NVRAM-based storage subsystem of claim 19 , wherein each slave controller is further configured to compare the measured number of occurrences of curable I/O errors to the threshold setting.
21. The NVRAM-based storage subsystem of claim 19 , wherein each slave controller is further configured to compare the measured error rate to the threshold setting.
22. The NVRAM-based storage subsystem of claim 15 , wherein the threshold setting relates to a statistical outlier, and wherein each slave controller is further configured to compare the measured performance characteristic to the threshold setting by determining whether the measured performance characteristic qualifies as a statistical outlier.
23. The NVRAM-based storage subsystem of claim 22 , wherein each slave controller is further configured to determine whether the measured performance characteristic qualifies as an outlier by determining whether measured performance characteristic is more than a pre-determined number of standard deviations away from a statistical mean of past instances of the measured performance characteristic.
24. The NVRAM-based storage subsystem of claim 15 , wherein each slave controller is further configured to adapt the threshold setting.
25. The NVRAM-based storage subsystem of claim 15 , wherein the master controller is further configured to:
summarize information encoding physical locations of defect regions on multiple NVRAM devices; and
store the summarized information for persistent storage.
26. The NVRAM-based storage subsystem of claim 25 , wherein the master controller is further configured to:
drive decisions managing subsequent I/O operations on the multiple NVRAM devices of the NVRAM-based storage subsystem in view of the summarized information stored.
27. The NVRAM-based storage subsystem of claim 15 , wherein the master controller is further configured to:
report, to the host computing device, the measured performance characteristic after the threshold setting has been surpassed; and
when reporting, summarize the measured performance characteristics.
28. The NVRAM-based storage subsystem of claim 27 , wherein the master controller is configured to:
cause the host computing device to upload the summarized information to a central server, wherein the central server is configured to use the summarized information to determine the threshold setting in subsequent manufacturing and deployment of NVRAM devices of the same type as the NVRAM devices of the NVRAM-based storage subsystem.
29. The NVRAM-based storage subsystem of claim 15 , wherein the master controller is further configured to:
project the reported information from a flash plane in which the performance characteristic was measured to a die that includes the flash plane, or to a package that includes the die.
30. A non-volatile random access memory (NVRAM)-based storage subsystem mounted on a host computing device, the NVRAM-based storage subsystem comprising:
a plurality of NVRAM devices, each comprising a plurality of packages and a slave controller, and each being configured to:
monitor an I/O operation on the NVRAM device by measuring a duration of completing the I/O operation on the NVRAM device and then comparing the measured duration of the I/O operation to a time-out value indicating an upper limit for the I/O operation on the NVRAM device beyond which the I/O operation on the NVRAM will be aborted;
responsive to determining that the measured duration of the monitored I/O operation exceeding the time-out value; identify at least one occurrence of error data associated with the monitored I/O operation;
measure a numeric aspect of the occurrence of the at least one error data on the NVRAM device, wherein the occurrence of the at least one error data is different from the duration of the particular I/O operation;
compare a numeric aspect of the occurrence of the at least one error data with a threshold setting; and
report the at least one occurrence of error data in response to the numeric aspect crossing the threshold setting;
a master controller means, in communication with at least one of the slave controllers, for determining a physical location of a defect on the NVRAM corresponding to the at least one error occurrence by analyzing the reported at least one error, wherein each package comprises multiple dice, and wherein each die comprises multiple planes.