Distributed storage system capable of restoring data in case of a storage failure

1. A collective data storage system comprising:

a plurality of storage nodes connected by a network, each node storing data as extents;

a data service (DS) agent in each node for managing the extents in the node;

a plurality of metadata service (MDS) agents for managing metadata relating to the nodes and the extents, the MDS agents operating in a subset of the nodes;

a cluster manager (CM) agent in each node for detecting a failure in the system and notifying a subset of the DS or MDS agents of the failure,

wherein upon notification of the failure, the subset of DS or MDS agents independently generates a plan to restore the extents affected by the failure and collectively restoring the affected extents based on the plan; and

a persistent map that correlates the data extents with the nodes; and wherein each MDS agent manages a subset of the map.

2. The storage system as recited in claim 1 further comprising an interface to allow a host application to access data stored in the system.

3. The storage system as in recited in claim 1 further comprising means for determining a subset of the nodes in which the MDS agents operate.

4. The storage system as recited in claim 1 , wherein each CM agent maintains an ordered list of the nodes that are currently operating in the system.

5. The storage system as recited in claim 1 , wherein each CM agent includes means for detecting a node failure.

6. The storage system as recited in claim 5 , wherein said means for detecting a node failure includes means for detecting a missing heart-beat within a predetermined time interval.

7. The storage system as recited in claim 1 , wherein each CM agent includes means for detecting a new node in the system.

8. The storage system as recited in claim 7 , wherein said means for detecting a new node includes means for monitoring network messages from the new node.

9. The storage system as recited in claim 1 , wherein each DS agent propagates updates to the extents in the associated node to other DS agents in the system.

10. The storage system as recited in claim 1 , wherein each DS agent manages data caching in the associated node.

11. The storage system as recited in claim 1 , wherein: each node comprises a plurality of data disks; and each DS agent includes means for detecting a failure of the disks in the associated node.

12. The storage system as recited in claim 11 , wherein said means for detecting the disk failure is based on error rates of the disks.

13. The storage system as recited in claim 1 , wherein the restore plan includes a list of the extents to be restored to re-establish data redundancy in the system.

14. The storage system as recited in claim 13 , wherein the extents to be restored are collectively restored by the DS agents.

15. The storage system as recited in claim 13 , wherein space is allocated in the nodes that are still operational to replace the extents affected by the failure.

16. The storage system as recited in claim 15 , wherein data in the affected extents is determined and transferred to the allocated space.

17. The storage system as recited in claim 1 , wherein: one of the MDS agents is notified of the failure; and the notified MDS agent determines those extents affected by the failure based on an Extent_To_Node map.

18. The storage system as recited in claim 1 , wherein: the DS agents are notified of the failure; and the notified DS agents determine those extents that have data on a failed node based on an Extent To Node map.

19. A method for restoring data in a collective storage system having a plurality of storage nodes interconnected by a network and storing data as extents, Data Service (DS) agents for managing the extents, Metadata Service (MDS) agents for managing metadata relating to the nodes and the extents, and Cluster Manager (CM) agents, the DS agents managing the extents in each node, the method comprising the steps of:

detecting a failure in the system by one of the CM agents;

notifying a subset of the DS or MDS agents of the failure;

generating, by the notified DS or MDS agents independently, a plan to restore the data extents affected by the failure; and

collectively restoring the affected extents based on the generated plan; wherein each node includes a plurality or disk drives; the detected failure is a disk failure; and the disk failure is detected by a DS agent based on error rates of the disks.

20. The method as recited in claim 19 , wherein the detected failure is a node failure.

21. The method as recited in claim 20 , wherein the node failure is detected based on a missing heart-beat within a predetermined time interval.

22. The method as recited in claim 19 , wherein the step of generating the restore plan includes the steps of; determining the extents affected by the failure; and allocating space in the nodes that are still operational to replace the affected extents.

23. The method as recited in claim 22 , wherein the step of restoring includes the steps of: determining data in the affected extents; and transferring the determined data to the allocated space.

24. The method as recited in claim 19 , wherein: each extent comprises a group of data fragments each residing on a different node, the DS agent associated with one of the fragments being a leader of the group; if the node on which the leader resides fails, then a new leader is chosen for the group as part of the data restoring; and the leader is notified of the fragments affected by the failure.

25. The method as recited in claim 19 , wherein each DS agent includes means for managing data caching and means for propagating updates on the data extents to other nodes.

26. The method as recited in claim 19 , wherein each DS agent includes means for propagating updates on the data extents of the DS agent to other nodes.

27. The method as recited in claim 19 further comprising the step of determining a subset of the nodes in which the MDS agents operate.

28. The method as recited in claim 19 further comprising the step of maintaining, by each CM agent, an ordered list of the nodes that are currently operating in the system.

29. The method as recited in claim 19 further comprising the step of detecting a node failure by a CM agent.

30. The method as recited in claim 29 , wherein the step of detecting a node failure includes the step of detecting a missing heart-beat within a predetermined time interval.

31. The method as recited in claim 19 further comprising the step of detecting a new node in the system.

32. The method as recited in claim 31 , wherein the step of detecting a new node includes the step of monitoring network messages from the new node.

33. The method as recited in claim 19 further comprising the step of propagating updates on the extents, by the DS agent in one of the nodes, to the DS agents in other nodes.

34. The method as recited in claim 19 , wherein the agents notified of the failure are the MDS agents.

35. The method as recited in claim 19 , wherein the agents notified of the failure are the DS agents.

36. A computer-program product for restoring data in a collective storage system having a plurality of storage nodes interconnected by a network and storing data as extents, Data Service (DS) agents for managing the extents, Metadata Service (MDS) agents for managing metadata relating to the nodes and the extents, and Cluster Manager (CM) agents, the DS agents managing the extents in each node, the computer-program product comprising a computer readable medium comprising program code for:

detecting a failure in the system by one of the CM agents; notifying a subset of the DS or MDS agents of the failure;

generating, by the notified DS or MDS agents independently, a plan to restore the extents affected by the failure;

collectively restoring the affected extents based on the generated plan;

wherein the generating the restore plan includes the steps of:

determining the extents affected by the failure;

allocating space in the nodes that are still operational to replace the affected extents; and

wherein the restoring includes the steps of:

determining data in the affected extents; and

transferring the determined data to the allocated space.