Rebuilding data across storage nodes
A method for proactively rebuilding user data in a plurality of storage nodes of a storage cluster is provided. The method includes distributing user data and metadata throughout the plurality of storage nodes such that the plurality of storage nodes can read the user data, using erasure coding, despite loss of two of the storage nodes. The method includes determining that one of the storage nodes is unreachable and determining to rebuild the user data for the one of the storage nodes that is unreachable. The method includes reading the user data across a remainder of the plurality of storage nodes, using the erasure coding and writing the user data across the remainder of the plurality of storage nodes, using the erasure coding. A plurality of storage nodes within a single chassis that can proactively rebuild the user data stored within the storage nodes is also provided.
1. A storage system, comprising:
a plurality of storage nodes, each storage node of the plurality of storage nodes having nonvolatile solid-state memory for user data storage; and
one or more controller of each of the plurality of storage nodes configured to write data across stripes in a first set of the plurality of storage nodes according to a first erasure coding scheme, the one or more controller configured to rebuild the data across stripes in a second set of the plurality of storage nodes responsive to a storage node of the first set of the plurality of storage nodes being unreachable.
2. The storage system of claim 1 , wherein the user data is rebuilt independent of a request from outside the storage cluster to read the user data, and wherein the plurality of storage nodes is configured to detect an unreachable storage node.
3. The storage system of claim 1 , further comprising:
each of the plurality of storage nodes is configured to emit a heartbeat, the heartbeat comprising a periodically issued message; and
each of the plurality of storage nodes is configured to detect the heartbeat of another of the plurality of storage nodes.
4. The storage system of claim 1 , wherein the plurality of storage nodes are distributed across multiple chassis.
5. The storage system of claim 1 , wherein:
the user data is striped across the first set of the plurality of storage nodes according to a RAID level, and the user data is rebuilt by reading and recovering the user data from the first set of the plurality of storage nodes according to the RAID level.
6. The storage system of claim 1 , further comprising:
each of the plurality of storage nodes is configured to consult others of the plurality of storage nodes to determine an erasure coding scheme for the user data that is striped across the first set of the plurality of storage nodes.
7. The storage cluster of claim 1 , further comprising:
each of the plurality of storage nodes configured to determine to which storage nodes to write recovered user data.
8. The storage system of claim 1 , wherein the storage nodes contain flash memory.
9. The storage system of claim 1 , wherein the data includes user data and meta data associated with the user data.
10. The storage system of claim 1 , wherein segments of the data are owned by differing authorities.