Non-disruptive and efficient migration of data across cloud providers
An index associates fingerprints of file segments to container numbers of containers within which the file segments are stored. At a start of migration, a boundary is created identifying a current container number. At least a subset of file segments at a source storage tier are packed into a new container to be written to a destination storage tier. A new container number is generated for the new container. The index is updated to associate fingerprints of the at least subset of file segments to the new container number. A request is received to read a file segment. The index is queried with a fingerprint of the file segment to determine whether the request should be directed to the source or destination storage tier based on a container number of a container within which the file segment is stored.
1. A method for migrating data of a deduplication file system from a source to destination storage tier comprising:
maintaining an index associating fingerprints of file segments to container numbers of containers within which the file segments are stored;
at a start of migration, creating a boundary identifying a current container number;
packing at least a subset of file segments at the source storage tier into a new container to be written to the destination storage tier;
generating a new container number for the new container;
updating the index to associate fingerprints of the at least subset of file segments to the new container number;
receiving a request to read a file segment; and
querying the index with a fingerprint of the file segment to determine whether the request should be directed to the source or destination storage tier based on a container number of a container within which the file segment is stored.
2. The method of claim 1 wherein container numbers are generated in a sequence, and the method further comprises:
in response to querying the index, receiving the container number of the container within which the file segment is stored;
comparing the container number to the boundary; and
directing the request to the source or destination storage tier based on the sequence, the sequence indicating an ordering of the container number with respect to the boundary.
3. The method of claim 1 wherein the source and destination storage tiers are within a same cloud provider.
4. The method of claim 1 wherein the source and destination storage tiers are provided by different cloud providers.
5. The method of claim 1 further comprising:
determining whether a file segment at the source storage tier is live or dead;
migrating live file segments from the source storage tier to the destination storage tier; and
not migrating dead file segments from the source storage tier to the destination storage tier.
6. The method of claim 1 wherein the deduplication file system is a log-structured deduplication file system in which the containers are maintained in a log and the method further comprises:
upon creating the boundary, migrating a first range of containers starting with a tail of the log and ending before the boundary;
writing a second range of containers comprising file segments from the first range of containers to a head of the log; and
updating the index to indicate that file segments associated with the first range of containers are now associated with the second range of containers.
7. A system for migrating data of a deduplication file system from a source to destination storage tier comprising: a processor; and memory configured to store one or more sequences of instructions which, when executed by the processor, cause the processor to carry out the steps of:
maintaining an index associating fingerprints of file segments to container numbers of containers within which the file segments are stored;
at a start of migration, creating a boundary identifying a current container number;
packing at least a subset of file segments at the source storage tier into a new container to be written to the destination storage tier;
generating a new container number for the new container;
updating the index to associate fingerprints of the at least subset of file segments to the new container number;
receiving a request to read a file segment; and
querying the index with a fingerprint of the file segment to determine whether the request should be directed to the source or destination storage tier based on a container number of a container within which the file segment is stored.
8. The system of claim 7 wherein container numbers are generated in a sequence, and the processor further carries out the steps of:
in response to querying the index, receiving the container number of the container within which the file segment is stored;
comparing the container number to the boundary; and
directing the request to the source or destination storage tier based on the sequence, the sequence indicating an ordering of the container number with respect to the boundary.
9. The system of claim 7 wherein the source and destination storage tiers are within a same cloud provider.
10. The system of claim 7 wherein the source and destination storage tiers are provided by different cloud providers.
11. The system of claim 7 wherein the processor further carries out the steps of:
determining whether a file segment at the source storage tier is live or dead;
migrating live file segments from the source storage tier to the destination storage tier; and
not migrating dead file segments from the source storage tier to the destination storage tier.
12. The system of claim 7 wherein the deduplication file system is a log-structured deduplication file system in which the containers are maintained in a log and the processor further carries out the steps of:
upon creating the boundary, migrating a first range of containers starting with a tail of the log and ending before the boundary;
writing a second range of containers comprising file segments from the first range of containers to a head of the log; and
updating the index to indicate that file segments associated with the first range of containers are now associated with the second range of containers.
13. A computer program product, comprising a non-transitory computer-readable medium having a computer-readable program code embodied therein, the computer-readable program code adapted to be executed by one or more processors to implement a method for migrating data of a deduplication file system from a source to destination storage tier comprising:
maintaining an index associating fingerprints of file segments to container numbers of containers within which the file segments are stored;
at a start of migration, creating a boundary identifying a current container number;
packing at least a subset of file segments at the source storage tier into a new container to be written to the destination storage tier;
generating a new container number for the new container;
updating the index to associate fingerprints of the at least subset of file segments to the new container number;
receiving a request to read a file segment; and
querying the index with a fingerprint of the file segment to determine whether the request should be directed to the source or destination storage tier based on a container number of a container within which the file segment is stored.
14. The computer program product of claim 13 wherein container numbers are generated in a sequence, and the method further comprises:
in response to querying the index, receiving the container number of the container within which the file segment is stored;
comparing the container number to the boundary; and
directing the request to the source or destination storage tier based on the sequence, the sequence indicating an ordering of the container number with respect to the boundary.
15. The computer program product of claim 13 wherein the source and destination storage tiers are within a same cloud provider.
16. The computer program product of claim 13 wherein the source and destination storage tiers are provided by different cloud providers.
17. The computer program product of claim 13 further comprising:
determining whether a file segment at the source storage tier is live or dead;
migrating live file segments from the source storage tier to the destination storage tier; and
not migrating dead file segments from the source storage tier to the destination storage tier.
18. The computer program product of claim 13 wherein the deduplication file system is a log-structured deduplication file system in which the containers are maintained in a log and the method further comprises:
upon creating the boundary, migrating a first range of containers starting with a tail of the log and ending before the boundary;
writing a second range of containers comprising file segments from the first range of containers to a head of the log; and
updating the index to indicate that file segments associated with the first range of containers are now associated with the second range of containers.