Data preservation for node evacuation in unstable nodes within a mesh
Described is a system and method that includes executing, by a processing device, a data risk mitigation service (DRMS), to monitor, for each node of a plurality of nodes in a mesh network, at least one of metadata indicating a health state of a node or a call for help from the node. The method includes determining, by the DRMS, whether the node is in an unstable state in view of the metadata or the call for help from the node. The method includes in response to determining that the node is in the unstable state, rerouting incoming data of the node and evacuating data stored in the node.
1. A method comprising:
executing, by a processing device, a data risk mitigation service (DRMS), to monitor, for each node of a plurality of nodes in a mesh network, at least one of metadata indicating a health state of a node or a call for help from the node;
determining, by the DRMS, whether the node is in an unstable state in view of the metadata or the call for help from the node; and
in response to determining that the node is in the unstable state, by the DRMS, rerouting incoming data of the node to a first destination and evacuating data stored in the node to a second destination, wherein each destination of the first destination and the second destination comprises at least one of a persistent volume, a cloud storage, or another node in the mesh network.
2. The method of claim 1 , wherein the metadata including a set of metrics, and wherein the set of metrics comprise a load on the node, a usage of a CPU of the node, a temperature of the node, a firmware version of the node, or a recent network connectivity issue.
3. The method of claim 2 , further comprising
determining, by the DRMS, that the node is in the unstable state in view of the set of metrics satisfying a first set of thresholds according to a first set of rules or receiving the call for help from the node.
4. The method of claim 3 , further comprising maintaining, by the DRMS, a lookup table including information of the node, the first destination of the rerouted data and the second destination of the evacuated data.
5. The method of claim 4 , further comprising
determining, by the DRMS, that the node reaches a stable state in view of the set of metrics satisfying a second set of thresholds according to a second set of rules; and
stopping the rerouting incoming data of the node and evacuating data stored in the node, in response to determining that the node reaches the stable state.
6. The method of claim 5 , further comprising
recovering the node by returning the rerouted data of the node and the evacuated data of the node to the node based on the lookup table.
7. The method of claim 1 , wherein the rerouting the incoming data of the node and evacuating the data stored in the node comprises deploying, by the DRMS, a script onto the node or using at least one of network rules or firewall rules to reroute the incoming data of the node and evacuate the data stored in the node.
8. A system comprising:
a memory;
a processing device, operatively coupled with the memory, to:
execute a data risk mitigation service (DRMS) to monitor, for each node of a plurality of nodes in a mesh network, at least one of metadata indicating a health state of a node or a call for help from the node;
determine whether the node is in an unstable state in view of the metadata or the call for help from the node; and
in response to determining that the node is in the unstable state, reroute incoming data of the node to a first destination and evacuate data stored in the node to a second destination, wherein each destination of the first destination and the second destination comprises at least one of a persistent volume, a cloud storage, or another node in the mesh network.
9. The system of claim 8 , wherein the metadata including a set of metrics, and wherein the set of metrics comprise a load on the node, a usage of a CPU of the node, a temperature of the node, a firmware version of the node, or a recent network connectivity issue.
10. The system of claim 9 , wherein the processing device is further to determine that the node is in the unstable state in view of the set of metrics satisfying a first set of thresholds according to a first set of rules or receiving the call for help from the node.
11. The system of claim 10 , wherein the processing device is further to maintain a lookup table including information of the node, the first destination of the rerouted data and the second destination of the evacuated data.
12. The system of claim 11 , wherein the processing device is further to
determine that the node reaches a stable state in view of the set of metrics satisfying a second set of thresholds according to a second set of rules;
stop the rerouting incoming data of the node and evacuating data stored in the node, in response to determining that the node reaches the stable state; and
recover the node by returning the rerouted data of the node and the evacuated data of the node to the node based on the lookup table.
13. The system of claim 8 , wherein the processing device is further to deploy a script onto the node or use at least one of network rules or firewall rules to reroute the incoming data of the node and evacuate the data stored in the node.
14. A non-transitory computer-readable storage medium including instructions that, when executed by a processing device, cause the processing device to:
execute a data risk mitigation service (DRMS) to monitor, for each node of a plurality of nodes in a mesh network, at least one of metadata indicating a health state of a node or a call for help from the node;
determine whether the node is in an unstable state in view of the metadata or the call for help from the node; and
in response to determining that the node is in the unstable state, reroute incoming data of the node to a first destination and evacuate data stored in the node to a second destination, wherein each destination of the first destination and the second destination comprises at least one of a persistent volume, a cloud storage, or another node in the mesh network.
15. The non-transitory computer-readable storage medium of claim 14 , wherein the metadata including a set of metrics, and wherein the set of metrics comprise a load on the node, a usage of a CPU of the node, a temperature of the node, a firmware version of the node, or a recent network connectivity issue.
16. The non-transitory computer-readable storage medium of claim 15 , wherein the processing device is further to
determine that the node is in the unstable state in view of the set of metrics satisfying a first set of thresholds according to a first set of rules or receiving the call for help from the node.
17. The non-transitory computer-readable storage medium of claim 16 , wherein the processing device is further to maintain a lookup table including information of the node, the first destination of the rerouted data and the second destination of the evacuated data.
18. The non-transitory computer-readable storage medium of claim 17 , wherein the processing device is further to
determine that the node reaches a stable state in view of the set of metrics satisfying a second set of thresholds according to a second set of rules;
stop the rerouting incoming data of the node and evacuating data stored in the node, in response to determining that the node reaches the stable state; and
recover the node by returning the rerouted data of the node and the evacuated data of the node to the node based on the lookup table.
19. The non-transitory computer-readable storage medium of claim 14 , wherein the processing device is further to deploy a script onto the node or use at least one of network rules or firewall rules to reroute the incoming data of the node and evacuate the data stored in the node.