Prioritization and quality-of-service control for near-zero-added-latency any point in time virtual replication
Prioritization and quality of service control operations for any PiT replication operations. When replicating data from a production site to a replica site, the transmission of data is prioritized based on the perspective of a splitter and based on the perspective of an appliance, which may coordinate with multiple splitters. Quality is controlled by prioritizing which data is sent. The transmission may also perform compression when transmitting the data and may handle situations where available resources are scarce or unavailable.
1. A method, comprising:
operating a splitter in a computing system, wherein the splitter is associated with consistency groups, wherein each of the consistency groups includes a group of devices that is replicated together, wherein the splitter is configured to split writes directed to the group of devices to an appliance such that the writes to the group of devices are replicated to a destination, wherein writes associated with different consistency groups are stored in corresponding journals at the splitter;
determining a priority for each consistency group associated with the splitter;
coordinating with the appliance to send an amount of data available in the journals of the splitter that is prepared for evacuation and to receive an amount of resources available at the appliance;
selecting data packets from the journals of the splitter based on the amount of resources available at the appliance and the priorities of the consistency groups; and
sending the selected data packets to the appliance.
2. The method of claim 1 , further comprising normalizing the priorities of the consistency groups such that a sum of the priorities equals 1.
3. The method of claim 1 , wherein the appliance coordinates with multiple appliances and each of the multiple appliances is associated with a splitter, further comprising:
determining a global priority for each of the consistency groups associated with the multiple appliances;
dividing the amount of resources available at the appliance between the splitters associated with the multiple appliances; and
sending an adjusted amount of resources to each of the splitters, wherein the adjusted amount of resources is different for each of the splitters and is based on the global priorities of the splitters.
4. The method of claim 1 , wherein the priority of each consistency group is based on a relationship between a current lag and a recovery point objective (RPO) of each consistency group.
5. The method of claim 4 , wherein the relationship includes a ratio between the current lag and the RPO or wherein the relationship includes a ration between the current lag squared and the RPO.
6. The method of claim 3 , further comprising adjusting the adjusted amount of resources sent to each of the splitters when one of the appliances cannot evacuate its own buffer.
7. The method of claim 3 , further comprising compressing the data of one or more of the consistency groups when compressing improves performance or reduces use of resources.
8. The method of claim 1 , further comprising changing a mode of operation of at least one consistency group when resources are unavailable to the splitter to a marking on host mode, wherein only metadata for the at least one consistency group is saved in the journal in the marking on host mode.
9. The method of claim 7 , further comprising compressing the data at rest.
10. The method of claim 1 , further comprising repeatedly evaluating the priority of each consistency group.
11. The method of claim 1 , wherein the selection of the data packet accounts for resources available to the splitter and resources available to the appliance.
12. A non-transitory storage medium having stored therein instructions that are executable by one or more hardware processors to perform operations comprising:
operating a splitter in a computing system, wherein the splitter is associated with consistency groups, wherein each of the consistency groups includes a group of devices that is replicated together, wherein the splitter is configured to split writes directed to the group of devices to an appliance such that the writes to the group of devices are replicated to a destination, wherein writes associated with different consistency groups are stored in corresponding journals at the splitter;
determining a priority for each consistency group associated with the splitter;
coordinating with the appliance to send an amount of data available in the journals of the splitter that is prepared for evacuation and to receive an amount of resources available at the appliance;
selecting data packets from the journals of the splitter based on the amount of resources available at the appliance and the priorities of the consistency groups, and
sending the selected data packets to the appliance.
13. The non-transitory storage medium of claim 12 , the operations further comprising normalizing the priorities of the consistency groups such that a sum of the priorities equals 1.
14. The non-transitory storage medium of claim 12 , wherein the appliance coordinates with multiple appliances and each of the multiple appliances is associated with a splitter, further comprising:
determining a global priority for each of the consistency groups associated with the multiple appliances;
dividing the amount of resources available at the appliance between the splitters associated with the multiple appliances; and
sending an adjusted amount of resources to each of the splitters, wherein the adjusted amount of resources is different for each of the splitters and is based on the global priorities of the splitters.
15. The non-transitory storage medium of claim 12 , wherein the priority of each consistency group is based on a relationship between a current lag and a recovery point objective (RPO) of each consistency group.
16. The non-transitory storage medium of claim 14 , the operations further comprising adjusting the adjusted amount of resources sent to each of the splitters when one of the appliances cannot evacuate its own buffer.
17. The non-transitory storage medium of claim 14 , the operations further comprising compressing the data of one or more of the consistency groups when compressing improves performance or reduces use of resources.
18. The non-transitory storage medium of claim 12 , the operations further comprising changing a mode of operation of at least one consistency group when resources are unavailable to the splitter to a marking on host mode, wherein only metadata for the at least one consistency group is saved in the journal in the marking on host mode.
19. The non-transitory storage medium of claim 17 , the operations further comprising compressing the data at rest.
20. The non-transitory storage medium of claim 12 , the operations further comprising repeatedly evaluating the priority of each consistency group.