System and method for optimized signature comparisons and data replication

1. A system comprising a source network node and a remote network node, the source network node having a processor and a memory storing instructions which, when executed by the processor, cause the processor to:

receive a command to duplicate a dataset located on the source network node to the remote network node, the dataset comprising signatures of data and metadata in a tree structured file system, wherein the data and metadata are stored as objects in an object store, each object having a content-derived signature;

in response to receipt of the command, access, on the source network node, signatures of the data and the metadata of the dataset in a previously stored snapshot of the dataset and the signatures of the data and the metadata of the dataset in a current snapshot of the dataset, wherein the signatures of the data and the metadata of the dataset in the previously stored snapshot are in a tree structured file system, and wherein the previously stored snapshot corresponds to a remote snapshot of the dataset previously stored at the remote network node;

perform, at the source network node, a local signature comparison between the previously stored snapshot and the current snapshot to identify one or more signatures that do not match between the previously stored snapshot and the current snapshot, wherein the local signature comparison is an iterative top-down comparison beginning at the root of a tree of the tree structured file system, and wherein the signatures that do not match have at least one of different content or different placement in the tree;

send the signatures that do not match to the remote network node;

receive, from the remote network node, a response of whether the signatures that do not match exist at the remote network node; and

based on a determination that the signatures that do not match do not exist at the remote network node, send data associated with the signatures that do not match to the remote network node for incorporation within a current remote snapshot.

2. The system of claim 1 , wherein the source network node and the remote network node communicate across a wide area network (WAN).

3. The system of claim 2 , wherein the signatures that do not match are sent asynchronously across the WAN.

4. The system of claim 2 , wherein the signatures that do not match are batched prior to sending across the WAN.

5. The system of claim 4 , wherein the batched signatures include signatures from different sibling nodes of the tree.

6. A method for generating a remote backup copy on a remote network node of a dataset located on a source network node, the method comprising:

receiving a command to duplicate the dataset located on the source network node to the remote network node, the dataset comprising signatures of data and metadata in a tree structured file system, wherein the data and metadata are stored as objects in an object store, each object having a content-derived signature;

in response to receipt of the command, accessing, on the source network node, signatures of the data and the metadata of the dataset in a previously stored snapshot of the dataset and the signatures of the data and the metadata of the dataset in a current snapshot of the dataset, wherein the signatures of the data and the metadata of the dataset in the previously stored snapshot are in a tree structured file system, and wherein the previously stored snapshot corresponds to a remote snapshot of the dataset previously stored at the remote network node;

performing, at the source network node, a local signature comparison between the previously stored snapshot and the current snapshot to identify one or more signatures that do not match between the previously stored snapshot and the current snapshot, wherein the local signature comparison is an iterative top-down comparison beginning at the root of a tree of the tree structured file system, and wherein the signatures that do not match have at least one of different content or different placement in the tree;

sending the signatures that do not match to the remote network node;

receiving, from the remote network node, a response of whether the signatures that do not match exist at the remote network node; and

based on a determination that the signatures that do not match do not exist at the remote network node, sending data associated with the signatures that do not match to the remote network node for incorporation within a current remote snapshot.

7. The method of claim 6 , wherein the source network node and the remote network node communicate across a wide area network (WAN).

8. The method of claim 7 , wherein the signatures that do not match are sent asynchronously across the WAN.

9. The method of claim 7 , further comprising:

batching the signatures that do not match prior to sending across the WAN.

10. The method of claim 9 , wherein the batched signatures include signatures from different sibling nodes of the tree.

11. A non-transitory computer-readable storage medium having stored therein instructions which, when executed by a processor, cause the processor to:

receive a command to duplicate a dataset located on a source network node to a remote network node, the dataset comprising signatures of data and metadata in a tree structured file system, wherein the data and metadata are stored as objects in an object store, each object having a content-derived signature;

in response to receipt of the command, access, on the source network node, signatures of the data and the metadata of the dataset in a previously stored snapshot of the dataset and the signatures of the data and the metadata of the dataset in a current snapshot of the dataset, wherein the signatures of the data and the metadata of the dataset in the previously stored snapshot are in a tree structured file system, and wherein the previously stored snapshot corresponds to a remote snapshot of the dataset previously stored at the remote network node;

perform, at the source network node, a local signature comparison between the previously stored snapshot and the current snapshot to identify one or more signatures that do not match between the previously stored snapshot and the current snapshot, wherein the local signature comparison is an iterative top-down comparison beginning at the root of a tree of the tree structured file system, and wherein the signatures that do not match have at least one of different content or different placement in the tree;

send the signatures that do not match to the remote network node;

receive, from the remote network node, a response of whether the signatures that do not match exist at the remote network node; and

based on a determination that the signatures that do not match do not exist at the remote network node, send data associated with the signatures that do not match to the remote network node for incorporation within a current remote snapshot.

12. The non-transitory computer-readable storage medium of claim 11 , wherein the source network node and the remote network node communicate across a wide area network (WAN).

13. The non-transitory computer-readable storage medium of claim 12 , wherein the signatures that do not match are sent asynchronously across the WAN.

14. The non-transitory computer-readable storage medium of claim 12 , storing additional instructions which, when executed by the processor, cause the processor to perform operations comprising:

batching the signatures that do not match prior to sending across the WAN.

15. The non-transitory computer-readable storage medium of claim 14 , wherein the batched signatures include signatures from different sibling nodes of the tree.

16. The system of claim 1 , wherein remote network node is to, based on receipt of the signatures that do not match from the source network node, perform a search for the signatures that do not match and send the response of whether the signatures that do not match exist at the remote network node to the source network node.

17. The system of claim 1 , wherein each of the content-derived signatures of the objects comprises a cryptographic digest of an entire content of an object.

18. The method of claim 6 , wherein each of the content-derived signatures of the objects comprises a cryptographic digest of an entire content of an object.

19. The method of claim 6 , wherein remote network node is to, based on receipt of the signatures that do not match from the source network node, perform a search for the signatures that do not match and send the response of whether the signatures that do not match exist at the remote network node to the source network node.

20. The non-transitory computer-readable storage medium of claim 11 , wherein each of the content-derived signatures of the objects comprises a cryptographic digest of an entire content of an object.