Storage system with asynchronous messaging between processing modules for data replication

1. An apparatus comprising:

a first storage system comprising a plurality of storage nodes;

the first storage system being configured to participate in a replication process with a second storage system;

each of the storage nodes of the first storage system comprising a plurality of storage devices;

each of the storage nodes of the first storage system further comprising a set of processing modules configured to communicate over one or more networks with corresponding sets of processing modules on other ones of the storage nodes;

the sets of processing modules of the storage nodes collectively comprising at least a portion of a distributed storage controller of the first storage system;

wherein in conjunction with the replication process, a first one of the processing modules is configured to generate a request message for transmission to a second one of the processing modules requesting that the second processing module transfer designated replication data to the second storage system;

the request message comprising a unique message identifier that the second processing module incorporates into a subsequent response message transmitted back to the first processing module to indicate a result of the requested transfer of the designated replication data to the second storage system;

the unique message identifier thereby relating the request message to the subsequent response message so as to support asynchronous messaging between the first and second processing modules for the requested transfer of the designated replication data within the replication process;

each of the sets of processing modules comprising a plurality of control modules and a plurality of routing modules;

the first processing module comprising a given one of the control modules;

the second processing module comprising a given one of the routing modules;

in conjunction with the replication process, the given control module is configured to generate the request message as a control-to-routing message for transmission to the given routing module requesting that the given routing module transfer the designated replication data to the second storage system;

the control-to-routing message comprising the unique message identifier that the given routing module incorporates into the subsequent response message as a routing-to-control message transmitted back to the given control module to indicate the result of the requested transfer of the designated replication data to the second storage system;

the unique message identifier thereby relating the control-to-routing message to the routing-to-control message so as to support asynchronous messaging between the given control module and the given routing module for the requested transfer of the designated replication data within the replication process;

wherein the given routing module is configured to initiate a particular instance of a background thread for processing the control-to-routing message received from the given control module; and

wherein each of the storage nodes is implemented using at least one processing device comprising a processor coupled to a memory.

2. The apparatus of claim 1 wherein the first and second storage systems comprise respective content addressable storage systems having respective sets of non-volatile memory storage devices.

3. The apparatus of claim 1 wherein the first and second storage systems are associated with respective source and target sites of the replication process and wherein the source site comprises a production site data center and the target site comprises a disaster recovery site data center.

4. The apparatus of claim 1 wherein the control-to-routing message and the routing-to-control message comprise respective synchronous messages of the first storage system that are separately acknowledged within a designated synchronous messaging timeout period by respective receiving ones of the given routing and control modules and further wherein an elapsed time between transmission of the control-to-routing message by the given control module to the given routing module and receipt of the corresponding routing-to-control message by the given control module from the given routing module is greater than the designated synchronous messaging timeout period.

5. The apparatus of claim 1 wherein the particular instance of the background thread for processing the control-to-routing message received from the given control module also generates the routing-to-control message for transmission back to the given control module.

6. The apparatus of claim 5 wherein the given routing module is configured to generate an acknowledgement of the control-to-routing message for transmission back to the given control module in conjunction with its initiation of the particular instance of the background thread.

7. The apparatus of claim 1 wherein the given control module is configured to store context information in conjunction with generating the control-to-routing message and further wherein the context information includes the unique message identifier.

8. The apparatus of claim 1 wherein the given control module is configured to initiate a particular instance of a background thread for processing the routing-to-control message received from the given routing module.

9. The apparatus of claim 8 wherein the given control module is configured to perform at least one of:

generate an acknowledgement of the routing-to-control message for transmission back to the given routing module in conjunction with its initiation of the particular instance of the background thread; and

retrieve stored context information comprising the unique message identifier in conjunction with its initiation of the particular instance of the background thread.

10. The apparatus of claim 1 wherein the unique message identifier is configured to maintain its uniqueness over a plurality of control-to-routing messages generated by the given control module and across one or more restarts of the given control module.

11. The apparatus of claim 1 wherein the control-to-routing message has a designated asynchronous messaging timeout period associated therewith and further wherein the given control module is configured to retry transmission of another control-to-routing message comprising a different unique message identifier if the routing-to-control message is not received from the given routing module within the designated asynchronous messaging timeout period.

12. The apparatus of claim 1 wherein the replication process comprises a cycle-based asynchronous replication process.

13. The apparatus of claim 1 wherein the unique message identifier is randomly generated or pseudorandomly generated.

14. A method comprising:

configuring a first storage system to include a plurality of storage nodes each having a plurality of storage devices, each of the storage nodes further comprising a set of processing modules configured to communicate over one or more networks with corresponding sets of processing modules on other ones of the storage nodes;

configuring the first storage system to participate in a replication process with a second storage system; and

in conjunction with the replication process, a first one of the processing modules generating a request message for transmission to a second one of the processing modules requesting that the second processing module transfer designated replication data to the second storage system;

the request message comprising a unique message identifier that the second processing module incorporates into a subsequent response message transmitted back to the first processing module to indicate a result of the requested transfer of the designated replication data to the second storage system;

the unique message identifier thereby relating the request message to the subsequent response message so as to support asynchronous messaging between the first and second processing modules for the requested transfer of the designated replication data within the replication process;

each of the sets of processing modules comprising a plurality of control modules and a plurality of routing modules;

the first processing module comprising a given one of the control modules;

the second processing module comprising a given one of the routing modules;

in conjunction with the replication process, the given control module is configured to generate the request message as a control-to-routing message for transmission to the given routing module requesting that the given routing module transfer the designated replication data to the second storage system;

the control-to-routing message comprising the unique message identifier that the given routing module incorporates into the subsequent response message as a routing-to-control message transmitted back to the given control module to indicate the result of the requested transfer of the designated replication data to the second storage system;

the unique message identifier thereby relating the control-to-routing message to the routing-to-control message so as to support asynchronous messaging between the given control module and the given routing module for the requested transfer of the designated replication data within the replication process;

wherein the given routing module is configured to initiate a particular instance of a background thread for processing the control-to-routing message received from the given control module; and

wherein the method is implemented by at least one processing device comprising a processor coupled to a memory.

15. The method of claim 14 wherein the control-to-routing message and the routing-to-control message comprise respective synchronous messages of the first storage system that are separately acknowledged within a designated synchronous messaging timeout period by respective receiving ones of the given routing and control modules and further wherein an elapsed time between transmission of the control-to-routing message by the given control module to the given routing module and receipt of the routing-to-control message by the given control module from the given routing module is greater than the designated synchronous messaging timeout period.

16. The method of claim 14 wherein the unique message identifier is randomly generated or pseudorandomly generated.

17. The method of claim 14 wherein the particular instance of the background thread for processing the control-to-routing message received from the given control module also generates the routing-to-control message for transmission back to the given control module.

18. A computer program product comprising a non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code when executed by at least one processing device causes said at least one processing device:

to configure a first storage system to include a plurality of storage nodes each having a plurality of storage devices, each of the storage nodes further comprising a set of processing modules configured to communicate over one or more networks with corresponding sets of processing modules on other ones of the storage nodes;

to configure the first storage system to participate in a replication process with a second storage system; and

in conjunction with the replication process, a first one of the processing modules being configured to generate a request message for transmission to a second one of the processing modules requesting that the second processing module transfer designated replication data to the second storage system;

the request message comprising a unique message identifier that the second processing module incorporates into a subsequent response message transmitted back to the first processing module to indicate a result of the requested transfer of the designated replication data to the second storage system;

the unique message identifier thereby relating the request message to the subsequent response message so as to support asynchronous messaging between the first and second processing modules for the requested transfer of the designated replication data within the replication process;

each of the sets of processing modules comprising a plurality of control modules and a plurality of routing modules;

the first processing module comprising a given one of the control modules;

the second processing module comprising a given one of the routing modules;

in conjunction with the replication process, the given control module is configured to generate the request message as a control-to-routing message for transmission to the given routing module requesting that the given routing module transfer designated replication data to the second storage system;

the control-to-routing message comprising the unique message identifier that the given routing module incorporates into the subsequent response message as a routing-to-control message transmitted back to the given control module to indicate the result of the requested transfer of the designated replication data to the second storage system;

the unique message identifier thereby relating the control-to-routing message to the routing-to-control message so as to support asynchronous messaging between the given control module and the given routing module for the requested transfer of the designated replication data within the replication process;

wherein the given routing module is configured to initiate a particular instance of a background thread for processing the control-to-routing message received from the given control module.

19. The computer program product of claim 18 wherein the control-to-routing message and the routing-to-control message comprise respective synchronous messages of the first storage system that are separately acknowledged within a designated synchronous messaging timeout period by respective receiving ones of the given routing and control modules and further wherein an elapsed time between transmission of the control-to-routing message by the given control module to the given routing module and receipt of the routing-to-control message by the given control module from the given routing module is greater than the designated synchronous messaging time out period.

20. The computer program product of claim 18 wherein the unique message identifier is randomly generated or pseudorandomly generated.