Recovery synchronization in a distributed storage system

1. A method for data storage, comprising:

sending write commands to a first storage node, for storing data in a data volume on the first storage node;

in response to a decision to create a copy of the data volume on a second storage node, (i) sending the write commands both to the first storage node and to the second storage node, and (ii) creating and sending from the first storage node to the second storage node a snapshot of the data volume, wherein the snapshot includes at least a write command having a specified serial number;

creating the copy of the data volume by the second storage node, by receiving the snapshot and creating the copy of the data volume from the received snapshot and the write commands that are received at the second storage node and whose serial numbers are larger than the specified serial number; and

sending from the second storage node to the first storage node a snapshot request that specifies the specified serial number, and discarding in the second storage node any received write commands whose serial numbers are smaller than or equal to the specified serial number.

2. A method for data storage, comprising:

sending write commands to a first storage node, for storing data in a data volume on the first storage node;

in response to a decision to create a copy of the data volume on a second storage node, (i) sending the write commands both to the first storage node and to the second storage node, and (ii) creating and sending from the first storage node to the second storage node a snapshot of the data volume, wherein the snapshot includes at least a write command having a specified serial number;

creating the copy of the data volume by the second storage node, by receiving the snapshot and creating the copy of the data volume from the received snapshot and the write commands that are received at the second storage node and whose serial numbers are larger than the specified serial number; and

detecting in the first storage node a write command that (i) is received after the snapshot has been created, and (ii) has a serial number smaller than the specified serial number, and requesting the client node to re-issue the write command with a new serial number.

3. A computing system, comprising:

a client node; and

at least a first storage node and a second storage node,

wherein the client node is configured to send write commands to the first storage node for storing data in a data volume on the first storage node, and wherein, in response to a decision to create a copy of the data volume on the second storage node, the client node is configured to send the write commands both to the first storage node and to the second storage node, the first storage node is configured to create and send to the second storage node a snapshot of the data volume, and the second storage node is configured to create the copy of the data volume by the second storage node, based on the snapshot and on one or more of the write commands received at the second storage node,

wherein the first storage node is configured to include in the snapshot at least a write command having a specified serial number, wherein the second storage node is configured to receive the snapshot and create the copy of the data volume from the received snapshot and the received write commands whose serial numbers are larger than the specified serial number,

and wherein the second storage node is configured to send to the first storage node a snapshot request that specifies the specified serial number, and to discard any received write commands whose serial numbers are smaller than or equal to the specified serial number.

4. A computing system, comprising:

a client node; and

at least a first storage node and a second storage node,

wherein the client node is configured to send write commands to the first storage node for storing data in a data volume on the first storage node, and wherein, in response to a decision to create a copy of the data volume on the second storage node, the client node is configured to send the write commands both to the first storage node and to the second storage node, the first storage node is configured to create and send to the second storage node a snapshot of the data volume, and the second storage node is configured to create the copy of the data volume by the second storage node, based on the snapshot and on one or more of the write commands received at the second storage node,

wherein the first storage node is configured to include in the snapshot at least a write command having a specified serial number, wherein the second storage node is configured to receive the snapshot and create the copy of the data volume from the received snapshot and the received write commands whose serial numbers are larger than the specified serial number,

and wherein the first storage node is configured to detect a write command that (i) is received after the snapshot has been created, and (ii) has a serial number smaller than the specified serial number, and to request the client node to re-issue the write command with a new serial number.

5. A storage node, comprising:

an interface for communicating with a client node and with an additional storage node over a computer network; and

a processor, which is configured to receive from the client node write commands having respective serial numbers for storing data in a data volume, to receive from the additional storage node a request to create a snapshot of the data volume that includes at least a write command having a specified serial number, and to create and send the snapshot to the additional storage node,

wherein the processor is configured to detect a write command that (i) is received after the snapshot has been created, and (ii) has a serial number larger than the specified serial number, and to request the client node to re-issue the write command with a new serial number.