Data size based replication
Techniques are provided for data size based replication. In an example, a replication daemon registers to receive notifications about data changes from a protocol driver. The replication daemon can maintain a counter for each of one or more replication policies, where a counter tracks how much data has changed that corresponds to a replication policy, since the prior replication for that replication policy. Where the daemon receives a notification for the protocol driver, it can determine whether the notification applies to any replication policy, and update any counters accordingly. Where an amount of data changed since the prior replication exceeds a threshold for a replication policy, a replication can be started. Upon successful completion of the replication, the corresponding counter can be reset.
1. A system, comprising:
a processor; and
a memory that stores executable instructions that, when executed by the processor, facilitate performance of operations, comprising:
registering with a protocol driver that processes data modifications on a file system of a first computer storage cluster to receive notifications about close-modified, rename, and delete operations performed via the protocol driver;
determining that data on the first computer storage cluster has been modified based on a first notification from the protocol driver;
in response to the determining that the data has been modified, determining that the data corresponds to a replication policy of the first computer storage cluster;
determining that at least a threshold amount of data has been modified that corresponds to the replication policy since performing a replication that corresponds to the replication policy; and
in response to performing the determining that at least the threshold amount of data has been modified, replicating at least a portion of data that corresponds to the replication policy to a second computer storage cluster.
2. The system of claim 1 , wherein the performing the registering with the protocol driver comprises:
providing the protocol driver with a callback function that the protocol driver calls to send notifications about data changes.
3. The system of claim 1 , wherein the operations further comprise:
maintaining a first counter that tracks an amount of modified data that corresponds to the replication policy;
incrementing the first counter by an amount of the data that has been modified;
wherein the performing the determining that at least the threshold amount of data has been modified comprises determining that a first value of the first counter is at least as large as a second value of the threshold amount of data; and
after successfully performing the replicating of at least the portion of data that corresponds to the replication policy, resetting the first counter.
4. The system of claim 3 , wherein the replication is a first replication, and wherein the operations further comprise:
in response to determining that performing a second replication is unsuccessful, determining not to reset a second counter that corresponds to the second replication.
5. The system of claim 1 , wherein the operations further comprise:
extracting a matched replication policy name corresponding to the data from a file path indicated by a notification received from a protocol driver, the matched replication policy name identifying the replication policy.
6. The system of claim 5 , wherein the operations further comprise:
incrementing a counter that corresponds to the replication policy by an amount of data modified indicated by the notification.
7. A method, comprising:
registering, by a system comprising a processor, with a protocol driver that processes data modifications on a file system of a first computer storage cluster to receive notifications about close-modified, rename, and delete operations performed via the protocol driver;
in response to the determining, by the system and based on a first notification received from the protocol driver, that data on a first computer storage cluster is modified, determining that the data corresponds to a replication policy of the first computer storage cluster;
determining, by the system, that at least a threshold amount of data has been modified that corresponds to the replication policy since performing a replication that corresponds to the replication policy; and
in response to performing the determining that at least the threshold amount of data has been modified, replicating, by the system, at least a portion of data that corresponds to the replication policy to a second computer storage cluster.
8. The method of claim 7 , wherein the performing the determining that the data on the first computer storage cluster is modified comprises:
receiving, by the system, a notification from the protocol driver indicating that the first data is modified.
9. The method of claim 8 , wherein the performing the determining that the data corresponds to the replication policy of the first computer storage cluster comprises:
determining, by the system, that a file path that is identified by the notification corresponds to the replication policy.
10. The method of claim 9 , wherein the replication policy is a first replication policy, and further comprising:
determining, by the system, that the file path that is identified by the notification corresponds to a second replication policy, wherein a first set of files of the first replication policy differs from a second set of files of the second replication policy.
11. The method of claim 9 , further comprising:
in response to performing the determining that the file path that is identified by the notification corresponds to the replication policy, determining, by the system, that a first file corresponding to the data is omitted from an exclude filter of the first replication policy.
12. The method of claim 9 , further comprising:
in response to performing the determining that the file path that is identified by the notification corresponds to the replication policy, determining, by the system, that a first file corresponding to the data is included in an include filter of the first replication policy.
13. A non-transitory computer-readable medium comprising instructions that, in response to execution, cause a system comprising a processor to perform operations, comprising:
registering with a protocol driver that processes data modifications on a file system of a first computer storage cluster to receive notifications about close-modified, rename, and delete operations performed via the protocol driver;
determining, based on a first notification received from the protocol driver, that at least a threshold amount of first data has been modified that corresponds to a replication policy since performing a replication that corresponds to the replication policy; and
in response to performing the determining that at least the threshold amount of the first data has been modified, replicating at least a portion of data that corresponds to the replication policy from a first computer storage cluster to a second computer storage cluster.
14. The non-transitory computer-readable medium of claim 13 , wherein notifications received from the protocol driver identify a file path corresponding to the first data that has been modified, an input/output type corresponding to the first data that has been modified, and an amount of modified data corresponding to the first data that has been modified.
15. The non-transitory computer-readable medium of claim 13 , wherein the performing the replicating of at least the portion of the first data that corresponds to the replication policy comprises:
sending an indication to perform the replicating to a daemon that performs replications.
16. The non-transitory computer-readable medium of claim 13 , wherein the replication policy is a first replication policy, and wherein the operations further comprise:
in response to determining that second data on the first computer storage cluster is modified, and that the second data does not correspond to the first replication policy or a second replication policy, discarding information associated with the second data being modified.
17. The method of claim 7 , wherein the performing the registering with the protocol driver comprises:
providing, by the system, the protocol driver with a callback function that the protocol driver calls to send notifications about data changes.
18. The method of claim 7 , further comprising:
maintaining, by the system, a first counter that tracks an amount of modified data that corresponds to the replication policy;
incrementing, by the system, the first counter by an amount of the data that has been modified;
wherein the performing the determining that at least the threshold amount of data has been modified comprises determining, by the system, that a first value of the first counter is at least as large as a second value of the threshold amount of data; and
after successfully performing the replicating of at least the portion of data that corresponds to the replication policy, resetting, by the system, the first counter.
19. The method of claim 18 , wherein the replication is a first replication, and further comprising:
in response to determining that performing a second replication is unsuccessful, determining, by the system, not to reset a second counter that corresponds to the second replication.
20. The non-transitory computer-readable medium of claim 13 , wherein the performing the registering with the protocol driver comprises:
providing the protocol driver with a callback function that the protocol driver calls to send notifications about data changes.