Method for efficient storage node replacement
Methods and apparatus are described for a horizontally scalable high performance object storage architecture. Metadata are completely decoupled from object storage. Instead of file names, users are given a locator when the object is uploaded and committed. Users can store the locator along with their own metadata or embed it directly in the static content. Clients can choose which storage nodes to store data on based on dynamic measures of node performance. Since there is no coupling among storage servers, performance can scale horizontally by adding more nodes. The decoupling also allows the front end services and storage to scale independently. High service availability is achieved by object-level synchronous replication and having no single point of failure. Failed nodes are rebuilt using copies of data in other nodes without taking the cluster offline. In addition to the replication, the ability to add or remove nodes on-line reduces maintenance-related service downtime.
1. A computer-implemented method for recreating a first storage node in a cluster comprising a plurality of storage nodes, the computer-implemented method being executed by one or more computing devices, the computer-implemented method comprising:
creating a replica chain for each object stored in the cluster, the created replica chain indicating selected storage nodes in the cluster storing replicas of the object, the created replica chain further identifying a storage address of the object on each selected storage node, wherein each selected storage node has a designated source node in the created replica chain;
identifying objects replicated on the first storage node with reference to other storage nodes of the plurality of storage nodes and without reference to the first storage node;
recreating each of the identified objects on a second storage node with reference to the other storage nodes and without reference to the first storage node, wherein each identified object and its corresponding storage address are copied from the designated source node for the object on the first storage node among the other storage nodes according to the created replica chain for the identified object; and
designating the second storage node as a replacement node for the first storage node.
2. The computer-implemented method of claim 1 wherein the identifying comprises reading a directory of objects on one or more storage nodes among the other storage nodes, wherein each of the one or more storage nodes is designated as a source node by the created replica chain of each object in the directory.
3. The computer-implemented method of claim 1 wherein the storage address of each recreated object on the second storage node matches the storage address of the corresponding object on the first storage node.
4. The computer-implemented method of claim 1 wherein the other storage nodes serve object requests from an application node during the recreating.
5. A system for recreating a first storage node in a cluster comprising a plurality of storage nodes, the system comprising one or more computing devices comprising a processor, a memory, and a network interface attached to a network, the one or more computing devices configured to:
create a replica chain for each object stored in the cluster, the created replica chain indicating selected storage nodes in the cluster storing replicas of the object, the created replica chain further identifying a storage address of the object on each selected storage node, wherein each selected storage node has a designated source node in the created replica chain;
identify objects replicated on the first storage node with reference to other storage nodes of the plurality of storage nodes in the cluster and without reference to the first storage node;
recreate each of the identified objects on a second storage node with reference to the other storage nodes and without reference to the first storage node, wherein each identified object and its corresponding storage address are copied from the designated source node for the object among the other storage nodes according to the created replica chain for the identified object; and
designate the second storage node as a replacement node for the first storage node.
6. The system of claim 5 further configured to identify objects replicated on the first storage node by reading a directory of objects on one or more storage nodes among the other storage nodes, wherein each of the one or more storage nodes is designated as a source node by the created replica chain of each object in the directory.
7. The system of claim 5 wherein the storage address of each recreated object on the second storage node matches the storage address of the corresponding object on the first storage node.
8. The system of claim 5 further configured to serve object requests from an application node with the other storage nodes during the recreating.
9. A computer program product for recreating a first storage node in a cluster comprising a plurality of storage nodes, the computer program product comprising at least one non-transitory computer-readable medium having computer instructions stored therein which are configured to cause a computing device to:
create a replica chain for each object stored in the cluster, the created replica chain indicating selected storage nodes in the cluster storing replicas of the object, the created replica chain further identifying a storage address of the object on each selected storage node, wherein each selected storage node has a designated source node in the created replica chain;
identify objects replicated on the first storage node with reference to other storage node of the plurality of the storage nodes in the cluster and without reference to the first storage node;
recreate each of the identified objects on a second storage node with reference to the other storage nodes and without reference to the first storage node, wherein each identified object and its corresponding storage address are copied from the designated source node for the object among the other storage nodes according to the created replica chain for the identified object; and
designate the second storage node as a replacement node for the first storage node.
10. The computer program product of claim 9 further configured to identify objects replicated on the first storage node by reading a directory of objects on one or more storage nodes among the other storage nodes, wherein each of the one or more storage nodes is designated as a source node by the created replica chain of each object in the directory.
11. The computer program product of claim 9 wherein the storage address of each recreated object on the second storage node matches the storage address of the corresponding object on the first storage node.
12. The computer program product of claim 9 further configured to serve object requests from an application node with the other storage nodes during the recreating.