Efficient live-migration of remotely accessed data
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for migrating data from a first storage system to a second storage system. That data is for a process (e.g., a virtual machine, application, or some other process) operating on a computer system that is separate from the first storage system and the second storage system. That data is stored according to data subsets that are each exclusive of each other. As each data set is migrated, access to the data set by the process is precluded, but access for the remaining data sets is not affected. Once the data migration is complete, access to the data set by the process is restored.
1. A method performed by a data processing system comprising a plurality of physical machines, the method comprising:
storing, in a first physical machine, data for a data processing process running on a second physical machine that is separate from the first physical machine, the storing data comprising storing the data according to a plurality of data subsets that are each exclusive of each other, wherein an address range for the data subset on the first physical machine is mapped to a respective logical address range for the data processing process;
enabling, for the data processing process, read and write access to the data by the data processing process according to the logical addressing;
determining that a first data subset stored on the first physical machine is to be migrated to a third physical machine separate from the first physical machine and the second physical machine, and in response:
blocking read and write access by the data processing process to the first data subset while maintaining read and write access by the data processing process to the other data subsets; and
migrating, from the first physical machine to the third physical machine, the first data subset to store the data subset on the third physical machine.
2. The method of claim 1 , wherein, in response to the migrating, the address range for the first data subset on the third physical machine is mapped to the respective logical address range for the data processing process.
3. The method of claim 1 , wherein the data processing process is implemented in a virtual machine.
4. The method of claim 3 , wherein:
the determination that the first data subset stored on the first physical machine is to be migrated to the third physical machine is performed by the first physical machine; and
blocking read and write access to the first data subset while maintaining read and write access to the other data subsets comprises blocking, by the first physical machine, read and write access by the virtual machine to the first data subset.
5. The method of claim 4 , further comprising:
providing, by the first physical machine, a notification to the data processing process of the migration of the first data subset to the third physical machine; and
wherein updating the mapping by logically mapping the address range for the first data subset on the third physical machine to the respective logical address range for the data processing process is done in response to the data processing process receiving the notification provided by the first physical machine.
6. The method of claim 4 , wherein the notification instructs the data processing process to not access the first data subset.
7. The method of claim 1 , wherein migrating, from the first physical machine to the third physical machine, the first data subset to store the data subset on the third physical machine comprises sending the first data subset from the first physical machine directly to the third physical machine.
8. The method of claim 1 , wherein:
the determination that the first data subset stored on the first physical machine is to be migrated to the third physical machine is performed by the data processing process; and
migrating, from the first physical machine to the third physical machine, the first data subset to store the data subset on the third physical machine comprises:
instructing, by the data processing process, the first physical machine to send the first data subset to the data processing process; and
sending, by the data processing process, the first data subset received from the first physical machine to the third physical machine.
9. The method of claim 1 , the data subsets are data blocks.
10. A storage machine in a distributed computing system, comprising:
a computer; and
a non-transitory data storage; and
a memory storing instructions executable by the compute and that upon such execution cause the storage machine to perform operations comprising:
storing, in the storage machine, data for a data processing process running on a physical machine that is separate from the storage machine, the storing data comprising storing the data according to a plurality of data subsets that are each exclusive of each other, wherein an address range for the data subset on the storage machine is mapped to a respective logical address range for the data processing process;
enabling, for the data processing process, read and write access to the data by the data processing process according to the logical addressing;
determining that a first data subset stored on the storage machine is to be migrated to a second storage machine separate from the storage machine and the physical machine, and in response:
blocking read and write access by the data processing process to the first data subset while maintaining read and write access by the data processing process to the other data subsets; and
migrating, from the storage machine to the second storage machine, the first data subset to store the data subset on the third physical machine.
11. The system of claim 10 , wherein, in response to the migrating, the address range for the first data subset on the second storage machine is mapped to the respective logical address range for the data processing process.
12. The system of claim 10 , wherein the data processing process is implemented in a virtual machine.
13. The system of claim 12 , wherein:
the determination that the first data subset stored on the storage machine is to be migrated to the second storage machine is performed by the storage machine; and
blocking read and write access to the first data subset while maintaining read and write access to the other data subsets comprises blocking, by the storage machine, read and write access by the data processing process to the first data subset.
14. The system of claim 13 , further comprising:
providing, by the storage machine, a notification to the data processing process of the migration of the first data subset to the second storage machine; and
wherein updating the mapping by logically mapping the address range for the first data subset on the second storage machine to the respective logical address range for the data processing process is done in response to the data processing process receiving the notification provided by the storage machine.
15. The system of claim 14 , wherein the notification instructs the data processing process to not access the first data subset.
16. The system of claim 10 , wherein migrating, from the storage machine to the second storage machine, the first data subset to store the data subset on the second storage machine comprises sending the first data subset from the storage machine directly to the second storage machine.
17. The system of claim 10 wherein:
the determination that the first data subset stored on the storage machine is to be migrated to the second storage machine is performed by the data processing process; and
migrating, from the storage machine to the second storage machine, the first data subset to store the data subset on the second storage machine comprises:
instructing, by the data processing process, the storage machine to send the first data subset to the data processing process; and
sending, by the data processing process, the first data subset received from the storage machine to the second storage machine.