Method and apparatus for managing data storage systems

1. A method for a system manager in a data storage system to move data from source logical disk (LD) regions to target LD regions (hereafter “the region moves”) while the data storage system remains online to a host device, the data storage system providing a virtual volume to the host device, the virtual volume being initially mapped to the source LD regions in one or more logical disks, the source LD regions being mapped to physical storage regions in one or more physical storage devices, the method comprising:

generating a task comprising a list of the region moves to process for transferring data from the source LD regions to the target LD regions; and,

processing the task, comprising:

selecting a region move from the task, the region move comprises a transfer of data from a source LD region to a target LD region;

determining if the region move would create excessive load on the corresponding physical storage devices so that the data storage system appears offline to the host device; and

when the region move would create excessive load:

determining if there is at least one region move in the task that has not been selected before;

when there is at least one region move in the task that has not been selected before, repeating said processing for a next region move in the task; and

when there is not at least one region move in the task that has not been selected before, waiting for another region move from the task that is currently being performed to finish and then repeating said processing for the next region move in the task.

2. The method of claim 1 , further comprising:

when the region move would not create excessive load, performing the region move by:

causing writes to the source LD region to be mirrored to the target LD region;

causing data in the source LD region to be copied to the target LD region; and

updating a mapping of the virtual volume from the source LD region to the target LD region.

3. The method of claim 2 , wherein:

said determining if the region move would create excessive load on the corresponding physical storage devices further comprises determining if one or more load values from one or more load counters for the physical storage devices are greater than one or more thresholds; and

the method further comprises incrementing the one or more load counters when the region move would not create excessive load.

4. The method of claim 3 , wherein the one or more load counters comprises:

a mirror load counter for a first physical storage device, the first physical storage device storing the target LD region, the mirror load counter tracking a first load caused by the writes to the source LD region being mirrored to the target LD region;

a first copy load counter for the first physical storage device, the first copy load counter tracking a second load to the first physical storage device caused by the data in the source LD region being copied to the target LD region; and

a second copy load counter for a second physical storage device, the second physical storage device storing the source LD region, the second copy load counter tracking a third load to the second physical storage device caused by the data in the source LD region being copied to the target LD region.

5. The method of claim 4 , after said causing the data in the source LD to be copied to the target LD region, further comprising:

determining if the data in the source LD region has been successfully copied to the target LD region; and

when the data in the source LD region has been successfully copied to the target LD region, decrementing the first and the second copy load counters.

6. The method of claim 5 , further comprising:

when the data in the source LD region has not been successfully copied to the target LD region:

decrementing the mirror load counter, the first copy load counter, and the second copy load counter;

stopping other region moves that are in the task; and

notifying a user that the task has failed.

7. The method of claim 4 , after said causing the data in the source LD region to be copied to the target LD region and prior to said updating a mapping of the virtual volume to the LD regions, further comprising:

enabling a write-through mode where writes are sent directly to the physical storage devices;

blocking reads and writes to the data storage system;

flushing dirty cache to the physical storage devices;

determining if said flushing the dirty cache exceeds a time limit so the data storage system appears offline to the host device;

when said flushing the dirty cache does not exceed the time limit:

performing said updating a mapping;

resuming the reads and writes to the data storage system.

8. The method of claim 7 , further comprising:

when said flushing the dirty cache exceeds the time limit:

determining if a number of tries to flush the dirty cache exceeds a threshold reference;

when the number of tries to flush the dirty cache exceeds the threshold reference:

resuming the reads and writes to the data storage system;

informing a user that the region move has failed;

when the number of tries to flush the dirty cache is below the threshold reference:

resuming the reads and writes to the data storage system;

waiting before repeating above steps starting from said blocking reads and writes to the data storage system.

9. The method of claim 7 , after said causing the data in the source LD to be copied to the target LD region, further comprising:

determining if there is a sufficient number of completed region moves;

when there is the sufficient number of completed region moves:

putting the data storage system in a write-through mode where writes are sent directly to the physical storage devices without caching;

after the dirty cache is reduced, performing steps from said blocking reads and writes to the data storage system to said resuming the reads and write to the data storage system;

after said resuming the reads and writes to the data storage system, exiting the data storage system from the write-through mode; and

decrementing the mirror load counter.

10. The method of claim 2 , after said causing writes to the source LD region to be mirrored to the target LD region and prior to said causing the data in the source LD to be copied to the target LD region, further comprising:

saving a number of outstanding writes to a source logical disk of the source LD region, the outstanding writes beginning prior to said causing writes to the source LD region to be mirrored to the target LD region;

incrementing a barrier ID so that the outstanding writes have an old barrier ID and new writes to the source logical disk have a new barrier ID;

determining if a completed write to the source logical disk has the old barrier ID or the new barrier ID;

when the completed write has the old barrier ID:

decrementing the saved number of the outstanding writes;

determining if the saved number of the outstanding writes is zero;

when the saved number of the outstanding writes is zero, performing above steps staffing from said causing the data in the source LD to be copied to the target LD region;

when the saved number of the outstanding writes is not zero, repeating above steps staffing from said determining if a completed write to the source logical disk has the old barrier ID or the new barrier ID.

11. A method for a system manager in a data storage system to non-disruptively move data from source logical disk (LD) regions to target LD regions (hereafter “the region moves”) while the data storage system remains online to a host device, the data storage system providing a virtual volume to the host device, the virtual volume being initially mapped to the source LD regions in one or more logical disks, the source LD regions being mapped to physical storage regions in one or more physical storage devices, the method comprising:

generating a task comprising a list of region moves to process for transferring data from the source LD regions to the target LD regions;

processing the task;

enabling a write-through mode where writes are sent directly to the physical storage devices without caching in order to reduce dirty cache;

determining if an amount of data in the dirty cache has become less than a threshold after said enabling a write-through mode; and

when the dirty cache becomes less than the threshold:

temporarily blocking reads and writes from the host device to the data storage system;

flushing the dirty cache to the physical storage devices;

determining if said flushing the dirty cache exceeds a time limit; and

when said flushing the dirty cache is below the time limit:

updating a mapping of the virtual volume from the source LD regions to the target LD regions; and

resuming the reads and writes from the host device to the data storage system.

12. The method of claim 11 , further comprising:

after said processing the task and prior to said enabling a write-through mode, determining if there is a sufficient number of completed region moves, wherein said enabling a write-through mode occurs only when there is the sufficient number of completed region moves; and

after said resuming the reads and writes to the data storage system, exiting the data storage system from the write-through mode.

13. The method of claim 12 , further comprising:

when said flushing the dirty cache exceeds the time limit:

determining if a number of tries to flush the dirty cache exceeds a threshold reference;

when the number of tries to flush the dirty cache exceeds the threshold reference:

resuming the reads and writes from the host device to the data storage system;

informing a user of that the region moves have failed;

when the number of tries to flush the dirty cache is below the threshold reference:

resuming the reads and writes from the host device to the data storage system;

waiting before repeating above steps starting from said putting the data storage system in the write-through mode.

14. An apparatus for managing a data storage system, the apparatus comprising:

a plurality of physical storage devices; and

a system manager configured to:

provide a virtual volume composed of a plurality of logical disk (LD) regions associated with physical storage regions of the physical storage devices;

generate a task comprising a list of region moves to process for transferring data from source LD regions to target LD regions;

selecting a region move from the task;

determine a load on the physical storage devices that would be involved in the region move;

process the region move by transferring data from a source LD region to a target LD region when the load of the region move would be below a predetermined threshold; and

selecting a next region move in the task and repeating said determine and said process for the next region move when the load of the region move would be above the predetermined threshold and there is at least one region move in the task that has not been selected before.

15. The apparatus of claim 14 , wherein for said transferring data from the source LD region to the target LD region, the system manager is configured to:

cause writes to the source LD region to be mirrored to the target LD region;

cause data in the source LD region to be copied to the target LD region; and

update a mapping of the virtual volume from the source LD region to the target LD region.

16. The apparatus of claim 14 , wherein when there is not at least one region move in the task that has not been selected before, the system manager is further configured to wait for another region move that is currently being performed to finish before said selecting a next region move and said repeating said determine and said process for the next region move in the task.

17. The apparatus of claim 14 , wherein for determining a load on the physical storage devices that would be involved in a region move being currently processed, the system manager is configured to:

determine whether one or more current load values from one or more load counters associated with the physical storage devices are greater than one or more thresholds; and

increment the one or more load counters when the one or more current load values are below the one or more thresholds.

18. The apparatus of claim 17 , wherein the one or more load counters comprises:

a mirror load counter associated with a first physical storage device that contains the target LD region, the mirror load counter tracking a first load caused by the writes to the source LD region being mirrored to the target LD region;

a first copy load counter associated with the first physical storage device, the first copy load counter tracking a second load to the first physical storage device caused by the data in the source LD region being copied to the target LD region; and

a second copy load counter associated with a second physical storage device that contains the source LD region, the second copy load counter tracking a third load to the second physical storage device caused by the data in the source LD region being copied to the target LD region.