Mechanism for managing a migration of data with mapped page and dirty page bitmap sections
A method for managing a live migration operation includes partitioning a first data structure into N sections of the first data structure, the first data structure indicating a location, associated with a source storage, having data to be copied to a target storage, and transferring less than all of the N sections of the first data structure to a migration server.
1. A method for managing a migration of data, the method comprising:
partitioning a first data structure into N sections of the first data structure, N being an integer greater than one, the first data structure indicating a location, associated with a source storage, having data to be copied to a target storage;
storing a copy of a first subset of the N sections of the first data structure in a first buffer having a capacity that is smaller than a size of the N sections, the copy of the first subset of the N sections representing a first state of the first subset, the first subset indicating a location of a first data write to the source storage, the N sections corresponding to N second buffers that are available for logging data writes, and the first buffer being an additional buffer to the N second buffers that is set aside from logging data writes; and
transferring the copy of the first subset of the N sections of the first data structure from the first buffer.
2. The method of claim 1 , further comprising:
partitioning a second data structure into N sections of the second data structure;
tracking, by using the second data structure, a location of a third data write to the source storage; and
initiating a transfer of a subset of the N sections of the second data structure to a migration server, one section of the subset of the N sections of the second data structure indicating the location of the third data write.
3. The method of claim 2 , further comprising:
storing a passive copy of one active section of the N sections of the second data structure in an N+1th section of a memory of a storage controller;
transferring the passive copy to the migration server; and
tracking a fourth data write to the source storage associated with the one active section of the N sections during or after the transferring of the passive copy to the migration server.
4. The method of claim 2 , further comprising run length encoding a metadata associated with the second data structure.
5. The method of claim 2 , wherein:
a first section of the subset of the N sections of the second data structure is selected for the transfer based on a section status indicator corresponding to the first section, the section status indicator indicating that a source data corresponding to the first section is to be copied to the target storage; and
the method further comprises:
transferring the section status indicator to the migration server; and
selecting, by the migration server, the first section for a source data copying operation based on the section status indicator.
6. The method of claim 5 , wherein:
the section status indicator comprises a first state indicating that a section of the N sections of the second data structure corresponds to a source data to be copied to the target storage, and a second state indicating that a section of the N sections of the second data structure is to be omitted from selection for the transfer;
the first data write occurred during or after a transferring of a previous data structure to the migration server; and
the second data write occurred during or after the transferring of the copy of the first subset of the N sections of the first data structure to the migration server.
7. The method of claim 1 , wherein:
the copy of the first subset of the N sections of the first data structure is selected for the transfer based on a section status indicator corresponding to the first subset, the section status indicator indicating that a source data corresponding to the first subset is to be copied to the target storage; and
the method further comprises transferring the section status indicator to a migration server.
8. The method of claim 7 , wherein:
the section status indicator comprises a first state indicating that a section of the N sections of the first data structure corresponds to a source data to be copied to the target storage, and a second state indicating that a section of the N sections of the first data structure is to be omitted from selection for the transfer; and
the first data write occurred during or after a transferring of a previous data structure to the migration server.
9. The method of claim 1 , further comprising run length encoding a metadata associated with the first data structure.
10. A storage device for managing a migration of data, the storage device being configured to:
partition a first data structure into N sections of the first data structure, N being an integer greater than one, the first data structure indicating a location, associated with a source storage, having data to be copied to a target storage;
perform a first iteration of transferring based on the first data structure, the first iteration comprising:
storing a copy of a first subset of the N sections of the first data structure in a first buffer having a capacity that is smaller than a size of the N sections, the copy of the first subset of the N sections representing a first state of the first subset, the first subset indicating a location of a first data write to the source storage, the N sections corresponding to N second buffers that are available for logging data writes, and the first buffer being an additional buffer to the N second buffers that is set aside from logging data writes; and
transferring the copy of the first subset of the N sections of the first data structure from the first buffer; and
perform a second iteration of transferring based on the first data structure, the second iteration comprising:
storing a copy of a second subset of the N sections of the first data structure in the first buffer, the second subset indicating a location of a second data write to the source storage; and
transferring the copy of the second subset of the N sections of the first data structure from the first buffer,
wherein an amount of data transferred from the first buffer based on the first iteration and the second iteration is greater than the capacity of the buffer.
11. The storage device of claim 10 , wherein the storage device is configured to:
partition a second data structure into N sections of the second data structure;
track, by using the second data structure, a location of a third data write to the source storage; and
initiate a transfer of a subset of the N sections of the second data structure to a migration server, one section of the subset of the N sections of the second data structure indicating the location of the third data write.
12. The storage device of claim 11 , wherein the storage device is configured to:
store a passive copy of one active section of the N sections of the second data structure in an N+1th section of a storage controller memory;
transfer the passive copy to the migration server; and
track a fourth data write to the source storage associated with the one active section of the N sections during or after the transfer of the passive copy to the migration server.
13. The storage device of claim 11 , wherein the storage device is configured to run length encode a metadata associated with the second data structure.
14. The storage device of claim 11 , wherein the storage device is configured to:
select a first section of the subset of the N sections of the second data structure for the transfer based on a section status indicator corresponding to the first section, the section status indicator indicating that a source data corresponding to the first section is to be copied to the target storage; and
transfer the section status indicator to the migration server.
15. The storage device of claim 14 , wherein:
the section status indicator comprises a first state indicating that a section of the N sections of the second data structure corresponds to a source data to be copied to the target storage, and a second state indicating that a section of the N sections of the second data structure is to be omitted from selection for the transfer;
the first data write occurred during or after a transferring of a previous data structure to the migration server; and
the second data write occurred during or after the transferring of the copy of the first subset of the N sections of the first data structure to the migration server.
16. The storage device of claim 11 , wherein the storage device is configured to:
select the copy of the first subset of the N sections of the first data structure for the transfer based on a section status indicator corresponding to the first subset, the section status indicator indicating that a source data corresponding to the first subset is to be copied to the target storage; and
transfer the section status indicator to a migration server.
17. The storage device of claim 16 , wherein the section status indicator comprises a first state indicating that a section of the N sections of the first data structure corresponds to a source data to be copied to the target storage, and a second state indicating that a section of the N sections of the first data structure is to be omitted from selection for the transfer; and
the first data write occurred during or after a transferring of a previous data structure to the migration server.
18. The storage device of claim 10 , wherein the storage device is configured to run length encode a metadata associated with the first data structure.
19. A system for managing a migration of data, the system comprising:
a migration server; and
a storage device, wherein the storage device is configured to:
partition a first data structure into N sections of the first data structure, N being an integer greater than one, the first data structure indicating a location, associated with a source storage, having data to be copied to a target storage;
store a copy of a first subset of the N sections of the first data structure in a first buffer having a capacity that is smaller than a size of the N sections, the copy of the first subset of the N sections representing a first state of the first subset, the first subset indicating a location of a first data write to the source storage, the N sections corresponding to N second buffers that are available for logging data writes, and the first buffer being an additional buffer to the N second buffers that is set aside from logging data writes; and
transfer the copy of the first subset of the N sections of the first data structure from the buffer.
20. The system of claim 19 , wherein the storage device is configured to:
partition a second data structure into N sections of the second data structure;
track, by using the second data structure, a location of a third data write to the source storage; and
initiate a transfer of a subset of the N sections of the second data structure to a migration server, one section of the subset of the N sections of the second data structure indicating the location of the third data write.