Applying a rate limit across a plurality of storage systems

1. A method of applying a rate limit across a plurality of storage systems, the method comprising:

determining a rate limit for paired storage systems, the rate limit specifying the cumulative amount of input/output (‘I/O’) operations that may be serviced by the paired storage systems during a predetermined period of time;

receiving, by a first storage system from a second storage system, an amount of I/O operations serviced by the second storage system during a previous predetermined period of time;

determining whether the amount of I/O operations serviced by the second storage system during the previous predetermined period of time is less than half of the rate limit for the paired storage systems;

responsive to determining that the amount of I/O operations serviced by the second storage system during the previous predetermined period of time is less than half of the rate limit for the paired storage systems, setting a local rate limit for a next predetermined period of time for the first storage system to the difference between the rate limit for the paired storage systems and the amount of I/O operations serviced by the second storage system during the previous predetermined period of time; and

responsive to determining that the amount of I/O operations serviced by the second storage system during the previous predetermined period of time is not less than half of the rate limit for the paired storage systems, setting a local rate limit for a next predetermined period of time for the first storage system to half of the rate limit for the paired storage systems.

2. The method of claim 1 further comprising:

determining a rate limit for paired storage systems, the rate limit specifying the cumulative amount of input/output (‘I/O’) operations that may be serviced by the paired storage systems during a predetermined period of time;

receiving, by the second storage system from the first storage system, an amount of I/O operations serviced by the first storage system during a previous predetermined period of time;

determining whether the amount of I/O operations serviced by the first storage system during a previous predetermined period of time is less than half of the rate limit for the paired storage systems;

responsive to determining that the amount of I/O operations serviced by the first storage system during the previous predetermined period of time is less than half of the rate limit for the paired storage systems, setting a local rate limit for a next predetermined period of time for the second storage system to the difference between the rate limit for the paired storage systems and the amount of I/O operations serviced by the first storage system during the previous predetermined period of time; and

responsive to determining that the amount of I/O operations serviced by the first storage system during the previous predetermined period of time is not less than half of the rate limit for the paired storage systems, setting a local rate limit for a next predetermined period of time for the second storage system to half of the rate limit for the paired storage systems.

3. The method of claim 2 further comprising:

sending, from the first storage system to the second storage system at a predetermined interval, information describing the amount of I/O operations serviced by the first storage system during a previous predetermined period of time; and

sending, from the second storage system to the first storage system at the predetermined interval, information describing an amount of I/O operations serviced by the second storage system during the previous predetermined period of time.

4. The method of claim 3 further comprising modifying the predetermined interval.

5. The method of claim 2 further comprising wherein the sum of the local rate limit for the first storage system and the local rate limit for the second storage system exceeds the rate limit specifying the cumulative amount of I/O operations that may be serviced by the paired storage systems during the predetermined period of time.

6. The method of claim 1 further comprising:

detecting that the first storage system has not received information describing the amount of I/O operations serviced by the second storage system during the previous predetermined period of time; and

setting the local rate limit for the next predetermined period of time for the first storage system to the rate limit for the paired storage systems.

7. The method of claim 1 further comprising, within a particular storage system, setting the rate limit for a first node within the particular storage system in dependence upon amount of I/O operations serviced by the second node within the particular storage system during a previous predetermined period of time and the local rate limit for the particular storage system.

8. An apparatus for applying a rate limit across a plurality of storage systems, the apparatus including a computer processor and a computer memory operatively coupled to the computer processor, the computer memory having disposed within it computer program instructions that, when executed by the computer processor, cause the apparatus to carry out the steps of:

determining a rate limit for paired storage systems, the rate limit specifying the cumulative amount of input/output (‘I/O’) operations that may be serviced by the paired storage systems during a predetermined period of time;

receiving, from a second storage system, an amount of I/O operations serviced by the second storage system during a previous predetermined period of time;

determining whether the amount of I/O operations serviced by the second storage system during the previous predetermined period of time is less than half of the rate limit for the paired storage systems;

responsive to determining that the amount of I/O operations serviced by the second storage system during the previous predetermined period of time is less than half of the rate limit for the paired storage systems, setting a local rate limit for a next predetermined period of time for the first storage system to the difference between the rate limit for the paired storage systems and the amount of I/O operations serviced by the second storage system during the previous predetermined period of time; and

responsive to determining that the amount of I/O operations serviced by the second storage system during the previous predetermined period of time is not less than half of the rate limit for the paired storage systems, setting a local rate limit for a next predetermined period of time for the first storage system to half of the rate limit for the paired storage systems.

9. The apparatus of claim 8 further comprising computer program instructions that, when executed by the computer processor, cause the apparatus to carry out the steps of:

receiving, from the first storage system, an amount of I/O operations serviced by the first storage system during a previous predetermined period of time;

determining whether the amount of I/O operations serviced by the first storage system during a previous predetermined period of time is less than half of the rate limit for the paired storage systems;

responsive to determining that the amount of I/O operations serviced by the first storage system during the previous predetermined period of time is less than half of the rate limit for the paired storage systems, setting a local rate limit for a next predetermined period of time for the second storage system to the difference between the rate limit for the paired storage systems and the amount of I/O operations serviced by the first storage system during the previous predetermined period of time; and

responsive to determining that the amount of I/O operations serviced by the first storage system during the previous predetermined period of time is not less than half of the rate limit for the paired storage systems, setting a local rate limit for a next predetermined period of time for the second storage system to half of the rate limit for the paired storage systems.

10. The apparatus of claim 8 , wherein the sum of the local rate limit for each storage system exceeds rate limit for the system of paired storage systems.

11. The apparatus of claim 8 further comprising computer program instructions that, when executed by the computer processor, cause the apparatus to carry out the step of:

detecting that information has not been received describing the amount of I/O operations serviced by a particular storage system during the previous predetermined period of time; and

assuming that the particular storage system did not service any I/O operations during the previous predetermined period of time.

12. The apparatus of claim 8 further comprising computer program instructions that, when executed by the computer processor, cause the apparatus to carry out the steps of, within a particular storage system, setting the rate limit for a first node within the particular storage system in dependence upon amount of I/O operations serviced by the a second node within the particular storage system during a previous predetermined period of time and the local rate limit for the particular storage system.

13. A system of paired storage systems, the system of paired storage systems including a plurality of storage systems, each storage system including a computer processor and a computer memory operatively coupled to the computer processor, the computer memory having disposed within it computer program instructions that, when executed by the computer processor, cause each storage system to carry out the steps of:

determining a rate limit for the system of paired storage systems, the rate limit specifying the cumulative amount of input/output (‘I/O’) operations that may be serviced by the storage systems during a predetermined period of time;

receiving a cumulative amount of I/O operations serviced by each of the other storage systems in the system during a previous predetermined period of time;

determining whether the cumulative amount of I/O operations serviced by each of the other storage systems in the system during the previous predetermined period of time is less than a configurable share of I/O bandwidth for the other storage systems in the system of paired storage systems;

responsive to determining that the cumulative amount of I/O operations serviced by each of the other storage systems in the system during the previous predetermined period of time is less than a configurable share of I/O bandwidth for the other storage systems in the system of paired storage systems, setting a local rate limit for a next predetermined period of time to the difference between the rate limit for the system of paired storage systems and the cumulative amount of I/O operations serviced by each of the other storage systems in the system during the previous predetermined period of time; and

responsive to determining that the cumulative amount of I/O operations serviced by each of the other storage systems in the system during the previous predetermined period of time of time is not less than the configurable share of I/O bandwidth for the other storage systems in the system of paired storage systems, setting a local rate limit for a next predetermined period of time to a predetermined share of I/O bandwidth for the storage system.

14. The system of paired storage systems of claim 13 , wherein each storage system further includes computer program instructions that, when executed by the computer processor, cause each storage system to carry out the step of sending, to at least one of the other storage systems at a predetermined interval, information describing the amount of I/O operations serviced by the storage system during a previous predetermined period of time.

15. The system of paired storage systems of claim 13 , wherein each storage system further includes computer program instructions that, when executed by the computer processor, cause each storage system to carry out the step of modifying the predetermined interval.

16. The system of paired storage systems of claim 13 , wherein the sum of the local rate limit for each storage system exceeds rate limit for the system of paired storage systems.

17. The system of paired storage systems of claim 13 , wherein each storage system further includes computer program instructions that, when executed by the computer processor, cause each storage system to carry out the steps of:

detecting that the storage system has not received information describing the amount of I/O operations serviced by at least one other storage system during the previous predetermined period of time; and

assuming that the at least one other storage system did not service any I/O operations during the previous predetermined period of time.

18. The system of paired storage systems of claim 13 , wherein the predetermined share of I/O bandwidth for the storage system is different for at least two of the storage systems.

19. The system of paired storage systems of claim 18 , wherein the configurable share of I/O bandwidth for the other storage systems in the system of paired storage systems is equal to the sum of the predetermined share of I/O bandwidth for each other storage system.

20. The system of paired storage systems of claim 13 wherein each storage system further comprises computer program instructions that, when executed by the computer processor, cause each storage system to carry out the step of setting the rate limit for a first node within the particular storage system in dependence upon amount of I/O operations serviced by the a second node within the particular storage system during a previous predetermined period of time and the local rate limit for the particular storage system.