Non-disruptive and efficient migration of data across cloud providers
Data containers are stored at a first cloud and include file segments managed by a deduplication file system. Metadata containers are written to a log, identified by a container ID generated sequentially, and include references to data containers. An index is maintained to map segment fingerprints to container IDs. Upon starting a migration to a second cloud, a checkpoint is created identifying a container ID at a head of the log. During migration, the index is updated to map fingerprints of migrated segments to new container IDs, and referencing new data containers having the migrated segments and written to the second cloud. A request to read a file is received. The index is examined to identify a container ID associated with a data container storing segments of the file. Based on the identified container ID and the checkpoint, the data container is accessed from the first or second cloud.
1. A method for migrating data of a deduplication file system comprising:
storing a plurality of containers comprising data and metadata containers, wherein the data containers are stored at a first cloud storage service and comprise segments of files managed by the file system, and wherein the metadata containers are written to a log, identified by a container ID generated sequentially, and comprise references to the data containers;
maintaining a fingerprint index to map fingerprints of the segments to container IDs;
upon receiving an indication to migrate from the first to a second cloud storage service, creating a checkpoint identifying a container ID at a current head of the log; and
during the migration,
updating the fingerprint index to map fingerprints of migrated segments to new container IDs, and referencing new data containers comprising the migrated segments, the new data containers having been written to the second cloud storage service;
receiving a request to read a file;
examining the fingerprint index to identify a container ID associated with a data container storing one or more segments of the file; and
based on the identified container ID and the checkpoint, accessing the data container from the first cloud storage service or the second cloud storage service in response to the read request.
2. The method of claim 1 wherein the accessing the data container in response to the read request comprises:
comparing the identified container ID with the checkpoint;
if the identified container ID is greater than the checkpoint, accessing the data container from the second cloud storage service; and
if the identified container ID is less than the checkpoint, accessing the data container from the first cloud storage service.
3. The method of claim 1 further comprising during the migration, identifying segments at the first cloud storage service that are live and segments at the first cloud storage service that are dead, the identification comprising:
creating a perfect hash vector from the fingerprint index;
enumerating a namespace of the deduplication file system to obtain fingerprints from the namespace, the obtained fingerprints corresponding to live segments at the first cloud storage service;
hashing the obtained fingerprints;
setting bits in the perfect hash vector corresponding to positions calculated from the hashing;
reading the metadata containers to obtain fingerprints stored in the metadata containers;
hashing the fingerprints stored in the metadata containers;
determining whether a hashed fingerprint from a metadata container maps to a position in the perfect hash vector;
if the hashed fingerprint from the metadata container maps to a position in the perfect hash vector, determining that a segment corresponding to the hashed fingerprint from the metadata container is live and should be migrated; and
if the hashed fingerprint from the metadata container does not map to a position in the perfect hash vector, determining that the segment corresponding to the hashed fingerprint from the metadata container is dead and should not be migrated.
4. The method of claim 1 further comprising during the migration, receiving a request to write data; and
writing the data to the second cloud storage service.
5. The method of claim 1 further comprising during the migration, ingesting a stream of data comprising first, second, and third segments;
deduplicating the first, second, and third segments using the fingerprint index;
based on the deduplication, storing the first segment at the second cloud storage service because the fingerprint index indicates that the first segment cannot be found at either the first or second cloud storage services;
not storing the second segment because the fingerprint index indicates that the second segment can be found at the first cloud storage service, the second segment not yet having been migrated to the second cloud storage service; and
not storing the third segment because the fingerprint index indicates that the third segment can be found at the second cloud storage service.
6. The method of claim 1 further comprising during the migration:
writing a new data container to the second cloud storage service, the new data container comprising live segments from the first cloud storage service and not comprising dead segments from the first cloud storage service;
creating a new metadata container, referencing the new data container, and having a new container ID generated sequentially with respect to the container ID at the current head of the log; and
writing the new metadata container to the log, the new metadata container now being at the current head of the log,
wherein the log comprises metadata containers referencing data containers at the first cloud storage service, and metadata containers referencing data containers at the second cloud storage service.
7. A system comprising: a processor; and memory configured to store one or more sequences of instructions which, when executed by the processor, cause the processor to carry out the steps of:
storing a plurality of containers comprising data and metadata containers, wherein the data containers are stored at a first cloud storage service and comprise segments of files managed by a deduplication file system, and wherein the metadata containers are written to a log, identified by a container ID generated sequentially, and comprise references to the data containers;
maintaining a fingerprint index to map fingerprints of the segments to container IDs;
upon receiving an indication to migrate from the first to a second cloud storage service, creating a checkpoint identifying a container ID at a current head of the log; and
during the migration,
updating the fingerprint index to map fingerprints of migrated segments to new container IDs, and referencing new data containers comprising the migrated segments, the new data containers having been written to the second cloud storage service;
receiving a request to read a file;
examining the fingerprint index to identify a container ID associated with a data container storing one or more segments of the file; and
based on the identified container ID and the checkpoint, accessing the data container from the first cloud storage service or the second cloud storage service in response to the read request.
8. The system of claim 7 wherein the accessing the data container in response to the read request comprises:
comparing the identified container ID with the checkpoint;
if the identified container ID is greater than the checkpoint, accessing the data container from the second cloud storage service; and
if the identified container ID is less than the checkpoint, accessing the data container from the first cloud storage service.
9. The system of claim 7 further comprising during the migration, identifying segments at the first cloud storage service that are live and segments at the first cloud storage service that are dead, the identification comprising:
creating a perfect hash vector from the fingerprint index;
enumerating a namespace of the deduplication file system to obtain fingerprints from the namespace, the obtained fingerprints corresponding to live segments at the first cloud storage service;
hashing the obtained fingerprints;
setting bits in the perfect hash vector corresponding to positions calculated from the hashing;
reading the metadata containers to obtain fingerprints stored in the metadata containers;
hashing the fingerprints stored in the metadata containers;
determining whether a hashed fingerprint from a metadata container maps to a position in the perfect hash vector;
if the hashed fingerprint from the metadata container maps to a position in the perfect hash vector, determining that a segment corresponding to the hashed fingerprint from the metadata container is live and should be migrated; and
if the hashed fingerprint from the metadata container does not map to a position in the perfect hash vector, determining that the segment corresponding to the hashed fingerprint from the metadata container is dead and should not be migrated.
10. The system of claim 7 further comprising during the migration, receiving a request to write data; and
writing the data to the second cloud storage service.
11. The system of claim 7 further comprising during the migration, ingesting a stream of data comprising first, second, and third segments;
deduplicating the first, second, and third segments using the fingerprint index;
based on the deduplication, storing the first segment at the second cloud storage service because the fingerprint index indicates that the first segment cannot be found at either the first or second cloud storage services;
not storing the second segment because the fingerprint index indicates that the second segment can be found at the first cloud storage service, the second segment not yet having been migrated to the second cloud storage service; and
not storing the third segment because the fingerprint index indicates that the third segment can be found at the second cloud storage service.
12. The system of claim 7 further comprising during the migration:
writing a new data container to the second cloud storage service, the new data container comprising live segments from the first cloud storage service and not comprising dead segments from the first cloud storage service;
creating a new metadata container, referencing the new data container, and having a new container ID generated sequentially with respect to the container ID at the current head of the log; and
writing the new metadata container to the log, the new metadata container now being at the current head of the log,
wherein the log comprises metadata containers referencing data containers at the first cloud storage service, and metadata containers referencing data containers at the second cloud storage service.
13. A computer program product, comprising a non-transitory computer-readable medium having a computer-readable program code embodied therein, the computer-readable program code adapted to be executed by one or more processors to implement a method comprising:
storing a plurality of containers comprising data and metadata containers, wherein the data containers are stored at a first cloud storage service and comprise segments of files managed by a deduplication file system, and wherein the metadata containers are written to a log, identified by a container ID generated sequentially, and comprise references to the data containers;
maintaining a fingerprint index to map fingerprints of the segments to container IDs;
upon receiving an indication to migrate from the first to a second cloud storage service, creating a checkpoint identifying a container ID at a current head of the log; and
during the migration,
updating the fingerprint index to map fingerprints of migrated segments to new container IDs, and referencing new data containers comprising the migrated segments, the new data containers having been written to the second cloud storage service;
receiving a request to read a file;
examining the fingerprint index to identify a container ID associated with a data container storing one or more segments of the file; and
based on the identified container ID and the checkpoint, accessing the data container from the first cloud storage service or the second cloud storage service in response to the read request.
14. The computer program product of claim 13 wherein the accessing the data container in response to the read request comprises:
comparing the identified container ID with the checkpoint;
if the identified container ID is greater than the checkpoint, accessing the data container from the second cloud storage service; and
if the identified container ID is less than the checkpoint, accessing the data container from the first cloud storage service.
15. The computer program product of claim 13 further comprising during the migration, identifying segments at the first cloud storage service that are live and segments at the first cloud storage service that are dead, the identification comprising:
creating a perfect hash vector from the fingerprint index;
enumerating a namespace of the deduplication file system to obtain fingerprints from the namespace, the obtained fingerprints corresponding to live segments at the first cloud storage service;
hashing the obtained fingerprints;
setting bits in the perfect hash vector corresponding to positions calculated from the hashing;
reading the metadata containers to obtain fingerprints stored in the metadata containers;
hashing the fingerprints stored in the metadata containers;
determining whether a hashed fingerprint from a metadata container maps to a position in the perfect hash vector;
if the hashed fingerprint from the metadata container maps to a position in the perfect hash vector, determining that a segment corresponding to the hashed fingerprint from the metadata container is live and should be migrated; and
if the hashed fingerprint from the metadata container does not map to a position in the perfect hash vector, determining that the segment corresponding to the hashed fingerprint from the metadata container is dead and should not be migrated.
16. The computer program product of claim 13 further comprising during the migration, receiving a request to write data; and
writing the data to the second cloud storage service.
17. The computer program product of claim 13 further comprising during the migration, ingesting a stream of data comprising first, second, and third segments;
deduplicating the first, second, and third segments using the fingerprint index;
based on the deduplication, storing the first segment at the second cloud storage service because the fingerprint index indicates that the first segment cannot be found at either the first or second cloud storage services;
not storing the second segment because the fingerprint index indicates that the second segment can be found at the first cloud storage service, the second segment not yet having been migrated to the second cloud storage service; and
not storing the third segment because the fingerprint index indicates that the third segment can be found at the second cloud storage service.
18. The computer program product of claim 13 further comprising during the migration:
writing a new data container to the second cloud storage service, the new data container comprising live segments from the first cloud storage service and not comprising dead segments from the first cloud storage service;
creating a new metadata container, referencing the new data container, and having a new container ID generated sequentially with respect to the container ID at the current head of the log; and
writing the new metadata container to the log, the new metadata container now being at the current head of the log,
wherein the log comprises metadata containers referencing data containers at the first cloud storage service, and metadata containers referencing data containers at the second cloud storage service.