Clustered storage system configured with decoupling of process restart from in-flight command execution

1. An apparatus comprising:

a clustered storage system comprising a plurality of nodes;

each of at least a subset of the nodes further comprising:

a processor coupled to a memory; and

a set of processing modules configured to communicate over one or more networks with corresponding sets of processing modules on other ones of the nodes;

in conjunction with a failure of a first instance of a process running on a given one of the nodes and a subsequent restart of a second instance of the process, at least one of the processing modules being configured:

to identify at least one transfer buffer command of the first instance of the process;

to identify a plurality of logically ordered commands of the first instance of the process; and

to provide distinct treatment of the transfer buffer command relative to treatment of the logically ordered commands in a manner that ensures that the restart of the second instance of the process is not delayed to await completion of the transfer buffer command or the logically ordered commands.

2. The apparatus of claim 1 wherein the sets of processing modules collectively comprise at least a portion of a distributed storage controller of the storage system.

3. The apparatus of claim 1 wherein the transfer buffer command of the first instance of the process utilizes at least one node buffer which is also subject to utilization by the second instance of the process.

4. The apparatus of claim 1 wherein the logically ordered commands of the first instance of the process require strict ordering in their logical flow.

5. The apparatus of claim 1 wherein providing distinct treatment of the transfer buffer command relative to treatment of the logically ordered commands comprises:

directing each of a plurality of other processes to identify any of its buffers that are in use by one or more transfer buffer commands of the first instance of the process; and

marking the identified buffers as ghost buffers;

wherein any buffer marked as a ghost buffer is not utilized by the other processes until such time as its marking as a ghost buffer is cleared.

6. The apparatus of claim 5 wherein the second instance of the process performs the directing.

7. The apparatus of claim 6 wherein the second instance of the process periodically repeats the directing and further wherein any buffer previously marked as a ghost buffer that is no longer in use by one or more transfer buffer commands of the first instance of the process has its marking as a ghost buffer cleared.

8. The apparatus of claim 1 wherein said at least one of the processing modules is further configured to associate unique connection identifiers with respective ones of the logically ordered commands of the first instance of the process.

9. The apparatus of claim 8 wherein each of a plurality of other processes maintains for each of a plurality of its connections a reference count which counts a number of logically ordered commands currently executing on that connection.

10. The apparatus of claim 1 wherein providing distinct treatment of the transfer buffer command relative to treatment of the logically ordered commands comprises:

issuing at least one connection identifier update command to each of a plurality of other processes indicating that one or more connections having unique connection identifiers associated with respective ones of the logically ordered commands of the first instance of the process have been invalidated; and

establishing new unique connection identifiers for association with respective ones of a plurality of logically ordered commands of the second instance of the process;

wherein only a single unique connection identifier is considered valid at any given time for its corresponding connection.

11. The apparatus of claim 9 wherein providing distinct treatment of the transfer buffer command relative to treatment of the logically ordered commands comprises:

marking each of the logically ordered commands of the first instance of the process with a barrier flag;

wherein a particular one of the plurality of other processes receiving a given one of the logically ordered commands marked with the barrier flag waits for its maintained reference counts to go to zero on any invalidated connections before executing the given logically ordered command.

12. The apparatus of claim 11 wherein the barrier flag serves to establish a barrier between logically ordered commands of the first instance of the process and logically ordered commands of the second instance of the process.

13. The apparatus of claim 1 wherein said at least one of the processing modules comprises at least one system-wide management module implemented on a corresponding one of the nodes.

14. The apparatus of claim 1 wherein the sets of processing modules of at least a subset of the nodes each comprise at least one control module and at least one data module, with different portions of a logical address space of the storage system being assigned to different ones of the control modules, each of the control modules being configured to direct received write requests to particular ones of the data modules for storage of corresponding pages in particular storage devices.

15. A method comprising:

configuring a clustered storage system to include a plurality of nodes, each of at least a subset of the nodes further comprising a set of processing modules configured to communicate over one or more networks with corresponding sets of processing modules on other ones of the nodes;

in conjunction with a failure of a first instance of a process running on a given one of the nodes and a subsequent restart of a second instance of the process, at least one of the processing modules:

identifying at least one transfer buffer command of the first instance of the process;

identifying a plurality of logically ordered commands of the first instance of the process; and

providing distinct treatment of the transfer buffer command relative to treatment of the logically ordered commands in a manner that ensures that the restart of the second instance of the process is not delayed to await completion of the transfer buffer command or the logically ordered commands;

wherein the method is performed by at least one processing device comprising a processor coupled to a memory.

16. The method of claim 15 wherein providing distinct treatment of the transfer buffer command relative to treatment of the logically ordered commands comprises:

directing each of a plurality of other processes to identify any of its buffers that are in use by one or more transfer buffer commands of the first instance of the process; and

marking the identified buffers as ghost buffers;

wherein any buffer marked as a ghost buffer is not utilized by the other processes until such time as its marking as a ghost buffer is cleared.

17. The method of claim 15 wherein providing distinct treatment of the transfer buffer command relative to treatment of the logically ordered commands comprises:

issuing at least one connection identifier update command to each of a plurality of other processes indicating that one or more connections having unique connection identifiers associated with respective ones of the logically ordered commands of the first instance of the process have been invalidated; and

establishing new unique connection identifiers for association with respective ones of a plurality of logically ordered commands of the second instance of the process;

wherein only a single unique connection identifier is considered valid at any given time for its corresponding connection.

18. A computer program product comprising a non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code when executed by at least one processing device causes said at least one processing device:

to configure a clustered storage system to include a plurality of nodes, each of at least a subset of the nodes further comprising a set of processing modules configured to communicate over one or more networks with corresponding sets of processing modules on other ones of the nodes;

in conjunction with a failure of a first instance of a process running on a given one of the nodes and a subsequent restart of a second instance of the process, at least one of the processing modules being configured:

to identify at least one transfer buffer command of the first instance of the process;

to identify a plurality of logically ordered commands of the first instance of the process; and

to provide distinct treatment of the transfer buffer command relative to treatment of the logically ordered commands in a manner that ensures that the restart of the second instance of the process is not delayed to await completion of the transfer buffer command or the logically ordered commands.

19. The computer program product of claim 18 wherein providing distinct treatment of the transfer buffer command relative to treatment of the logically ordered commands comprises:

directing each of a plurality of other processes to identify any of its buffers that are in use by one or more transfer buffer commands of the first instance of the process; and

marking the identified buffers as ghost buffers;

wherein any buffer marked as a ghost buffer is not utilized by the other processes until such time as its marking as a ghost buffer is cleared.

20. The computer program product of claim 18 wherein providing distinct treatment of the transfer buffer command relative to treatment of the logically ordered commands comprises:

issuing at least one connection identifier update command to each of a plurality of other processes indicating that one or more connections having unique connection identifiers associated with respective ones of the logically ordered commands of the first instance of the process have been invalidated; and

establishing new unique connection identifiers for association with respective ones of a plurality of logically ordered commands of the second instance of the process;

wherein only a single unique connection identifier is considered valid at any given time for its corresponding connection.