Smooth restart of storage clusters in a storage system
Described embodiments provide methods and systems for smoothly restarting a crashed storage cluster application associated with a storage cluster of a storage system. A survivor process associated with each storage cluster application is started. If a crash of at least one of the storage cluster applications is detected, the storage system performs a smooth restart of the at least one crashed storage cluster application. A smooth restart of the at least one crashed storage cluster application is performed according to a new storage cluster application that is configured to replace the crashed storage cluster application by managing open handles associated with the crashed storage cluster application.
1. A method comprising:
starting a survivor process associated with a storage cluster application;
detecting a crash of the storage cluster application; and
performing a smooth restart of the storage cluster application after the crash according to a new storage cluster application configured to replace the storage cluster application by managing open handles associated with the storage cluster application,
wherein performing a smooth restart of the storage cluster application after the crash comprises:
recovering context data associated with the storage cluster application;
determining consistency of the recovered context data;
recovering open handles associated with the storage cluster application from the associated survivor process; and
replaying any commands that were not complete, based upon the recovered open handles when the crash was detected to the storage cluster application.
2. The method of claim 1 , further comprising:
setting an in progress attribute at times an associated command is started by the storage cluster application; and
clearing the in progress attribute at times the associated command is completed by the storage cluster application;
wherein replaying any commands that were not complete when the crash was detected to the storage cluster application comprises:
determining whether the in progress attribute is set for any commands; and
replaying any commands for which the in progress attribute is set.
3. The method of claim 1 , further comprising:
identifying one or more critical portions of commands processed by the storage cluster application;
upon reaching a critical portion of a command, incrementing a critical portion counter; and
upon completing a critical portion of a command, decrementing the critical portion counter;
wherein determining consistency of the recovered context data comprises determining that the recovered context data is consistent when the critical portion counter is equal to an expected value.
4. The method of claim 1 , further comprising:
opening one or more handles associated with a connection between a client and an associated storage cluster application;
incrementing a reference counter associated with the one or more open handles;
updating context data associated with the one or more open handles associated with the storage cluster application; and
passing the one or more open handles to the survivor process associated with the storage cluster application.
5. The method of claim 4 , further comprising incrementing the reference counter associated with the one or more open handles passed to the survivor process.
6. The method of claim 5 , further comprising decrementing the reference counter associated with the one or more open handles if a storage cluster application is stopped.
7. The method of claim 6 , further comprising:
closing, at times the reference counter associated with one of the one or more open handles is equal to zero, a corresponding handle;
removing context data associated with the closed handle; and
removing the closed handle from the survivor process associated with the storage cluster application.
8. A system comprising:
a processor; and
memory storing computer program code that when executed on the processor causes the processor to execute commands of a storage system operable to perform the operations of:
starting a survivor process associated with a storage cluster application;
detecting a crash of the storage cluster application; and
performing a smooth restart of the storage cluster application after the crash according to a new storage cluster application configured to replace the storage cluster application by managing open handles associated with the storage cluster application,
wherein performing a smooth restart of the storage cluster application after the crash comprises:
recovering context data associated with the storage cluster application;
determining consistency of the recovered context data;
recovering open handles associated with the storage cluster application from the associated survivor process; and
replaying any commands that were not complete, based upon the recovered open handles, when the crash was detected to the storage cluster application.
9. The system of claim 8 , wherein the system is further operable to perform the operations of:
setting an in progress attribute at times an associated command is started by the storage cluster application; and
clearing the in progress attribute at times the associated command is completed by the storage cluster application;
wherein replaying any commands that were not complete when the crash was detected to the storage cluster application comprises:
determining whether the in progress attribute is set for any commands; and
replaying any commands for which the in progress attribute is set.
10. The system of claim 8 , wherein the system is further operable to perform the operations of:
identifying one or more critical portions of commands processed by the storage cluster application;
upon reaching a critical portion of a command, incrementing a critical portion counter; and
upon completing a critical portion of a command, decrementing the critical portion counter;
wherein determining consistency of the recovered context data comprises determining that the recovered context data is consistent when the critical portion counter is equal to an expected value.
11. The system of claim 8 , wherein the system is further operable to perform the operations of:
opening one or more handles associated with a connection between a client and an associated storage cluster application;
incrementing a reference counter associated with the one or more open handles;
updating context data associated with the one or more open handles associated with the storage cluster application; and
passing the one or more open handles to the survivor process associated with the storage cluster application.
12. The system of claim 11 , wherein the system is further operable to perform the operations of:
incrementing a reference counter associated with the one or more open handles passed to the survivor process; and
decrementing the reference counter associated with the one or more open handles if a storage cluster application is stopped.
13. The system of claim 12 , wherein the system is further operable to perform the operations of:
closing, at times the reference counter associated with one of the one or more open handles is equal to zero, a corresponding handle;
removing context data associated with the closed handle; and
removing the closed handle from the survivor process associated with the storage cluster application.
14. A computer program product including a non-transitory computer readable storage medium having computer program code encoded thereon that when executed on a processor of a computer causes the computer to execute commands of a storage system, the computer program product comprising:
computer program code for starting a survivor process associated with a storage cluster application;
computer program code for detecting a crash of the storage cluster application; and
computer program code for performing a smooth restart of the storage cluster application after the crash according to a new storage cluster application configured to replace the storage cluster application by managing open handles associated with the storage cluster application,
wherein performing a smooth restart of the storage cluster application after the crash comprises:
recovering context data associated with the storage cluster application;
determining consistency of the recovered context data;
recovering open handles associated with the storage cluster application from the associated survivor process; and
replaying any commands that were not complete based upon the recovered open handles when the crash was detected to the storage cluster application.
15. The computer program product of claim 14 , further comprising:
computer program code for setting an in progress attribute at times an associated command is started by the storage cluster application; and
computer program code for clearing the in progress attribute at times the associated command is completed by the storage cluster application;
wherein replaying any commands that were not complete at times the crash was detected to the storage cluster application comprises:
determining whether the in progress attribute is set for any commands; and
replaying any commands for which the in progress attribute is set.
16. The computer program product of claim 15 , wherein the storage cluster application is further operable to perform the operations of:
computer program code for identifying one or more critical portions of commands processed by the storage cluster application;
computer program code for incrementing, upon reaching a critical portion of a command, a critical portion counter; and
computer program code for decrementing, upon completing a critical portion of a command, the critical portion counter;
wherein determining consistency of the recovered context data comprises determining that the recovered context data is consistent when the critical portion counter is equal to an expected value.
17. The computer program product of claim 16 , further comprising:
computer program code for opening one or more handles associated with a connection between a client and an associated storage cluster application;
computer program code for incrementing a reference counter associated with the one or more open handles;
computer program code for updating context data associated with the one or more open handles associated with the storage cluster application;
computer program code for passing the one or more open handles to the survivor process associated with the storage cluster application;
computer program code for incrementing a reference counter associated with the one or more open handles passed to the survivor process;
computer program code for decrementing the reference counter associated with the one or more open handles if a storage cluster application is stopped;
computer program code for closing, at times the reference counter associated with one of the one or more open handles is equal to zero, a corresponding handle;
computer program code for removing context data associated with the closed handle; and
computer program code for removing the closed handle from the survivor process associated with the storage cluster application.