Method and system for managing a spare persistent storage device and a spare node in a multi-node data cluster
In general, the invention relates to a method for managing data. The method includes detecting a failure of a persistent storage device in a data node of a plurality of data nodes, and in response to the detecting, initiating a rebuilding of data in a spare location using proactive copy metadata and slice metadata, wherein the data is a copy of data that was stored in the persistent storage device.
1. A method for managing data, the method comprising:
detecting, by a data processor, a failure of a persistent storage device in a data node of a plurality of data nodes; and
in response to the detecting, initiating, by the data processor, a rebuilding of data in a spare location using proactive copy metadata and slice metadata, wherein the data is a copy of data that was stored in the persistent storage device;
prior to detecting the failure of the persistent storage device in the data node of the plurality of data nodes:
detecting a potential failure of the persistent storage device; wherein detecting the potential failure of the persistent storage device comprises;
monitoring, by a health monitor associated with the persistent storage device, threshold values of attributes associated with the persistent storage device that indicate reliability of the persistent storage device;
making a determination, based on the monitoring and by the health monitor, of the potential failure of the persistent storage device; and
in response to the determination by the health monitor:
obtaining, by the data processor, a notification indicating the potential failure of the persistent storage device from the health monitor; and
in response to detecting the potential failure of the persistent storage device:
initiating, by the data processor, writing of proactive copy data to the spare location.
2. The method of claim 1 , wherein the spare location is a spare persistent storage device in the data node.
3. The method of claim 1 , wherein the spare location is a spare persistent storage device in a spare data node, wherein the spare data node is not one of the plurality of data nodes.
4. The method of claim 1 , wherein the proactive copy data comprises a copy of at least a portion of the data on the persistent storage device.
5. The method of claim 4 , wherein the proactive copy metadata is based on the proactive copy data.
6. The method of claim 4 , wherein the proactive copy data comprises at least one data chunk and at least one parity chunk.
7. A system, comprising:
a processor;
a data processor, which when executed by the processor performs a method, the method comprising:
detecting, by the data processor, a failure of a persistent storage device in a data node of a plurality of data nodes; and
in response to the detecting, initiating, by the data processor, a rebuilding of data in a spare location using proactive copy metadata and slice metadata, wherein the data is a copy of data that was stored in the persistent storage device;
prior to detecting the failure of the persistent storage device in the data node of the plurality of data nodes:
detecting a potential failure of the persistent storage device; wherein detecting the potential failure of the persistent storage device comprises:
monitoring, by a health monitor associated with the persistent storage device, threshold values of attributes associated with the persistent storage device that indicate reliability of the persistent storage device;
making a determination, based on the monitoring and by the health monitor, of the potential failure of the persistent storage device; and
in response to the determination by the health monitor:
obtaining, by the data processor, a notification indicating the potential failure of the persistent storage device from the health monitor; and
in response to detecting the potential failure of the persistent storage device:
initiating, by the data processor, writing of proactive copy data to the spare location.
8. The system of claim 7 , wherein the spare location is a spare persistent storage device in the data node.
9. The system of claim 7 , wherein the spare location is a spare persistent storage device in a spare data node, wherein the spare data node is not one of the plurality of data nodes.
10. The system of claim 7 , wherein the proactive copy data comprises a copy of at least a portion of the data on the persistent storage device.
11. The system of claim 10 , wherein the proactive copy metadata is based on the proactive copy data.
12. The system of claim 10 , wherein the proactive copy data comprises at least one data chunk and at least one parity chunk.
13. A non-transitory computer readable medium comprising computer readable program code, which when executed by a computer processor enables the computer processor to perform a method, the method comprising:
detecting, by a data processor, a failure of a persistent storage device in a data node of a plurality of data nodes; and
in response to the detecting, initiating, by the data processor, a rebuilding of data in a spare location using proactive copy metadata and slice metadata, wherein the data is a copy of data that was stored in the persistent storage device;
prior to detecting the failure of the persistent storage device in the data node of the plurality of data nodes:
detecting a potential failure of the persistent storage device; wherein detecting the potential failure of the persistent storage device comprises:
monitoring, by a health monitor associated with the persistent storage device, threshold values of attributes associated with the persistent storage device that indicate reliability of the persistent storage device;
making a determination, based on the monitoring and by the health monitor, of the potential failure of the persistent storage device; and
in response to the determination by the health monitor:
obtaining, by the data processor, a notification indicating the potential failure of the persistent storage device from the health monitor; and
in response to detecting the potential failure of the persistent storage device:
initiating, by the data processor, writing of proactive copy data to the spare location.
14. The non-transitory computer readable medium of claim 13 , wherein the spare location is a spare persistent storage device in the data node.
15. The non-transitory computer readable medium of claim 13 , wherein the spare location is a spare persistent storage device in a spare data node, wherein the spare data node is not one of the plurality of data nodes.
16. The non-transitory computer readable medium of claim 13 , wherein the proactive copy data comprises a copy of at least a portion of the data on the persistent storage device.
17. The non-transitory computer readable medium of claim 16 , wherein the proactive copy metadata is based on the proactive copy data.