Resume support for cloud storage operations
A method comprises storing information corresponding to a plurality of files being written from a source storage device to a cloud storage platform in a file transfer operation. The method further comprises, in response to a failure of the file transfer operation, identifying from the information one or more file offsets successfully written to the cloud storage platform for one or more of the plurality of the files prior to the failure. The file transfer operation for the one or more of the plurality of files is resumed from the identified one or more file offsets.
1. An apparatus comprising:
at least one processing platform comprising at least one processor coupled to a memory;
said at least one processing platform being configured:
to store information corresponding to a plurality of files being written from a source storage device to a cloud storage platform in a file transfer operation, wherein the information corresponding to the plurality of files is stored in one or more databases;
in response to a failure of the file transfer operation, to identify from the information one or more file offsets successfully written to the cloud storage platform for one or more of the plurality of the files prior to the failure;
to cause resumption of the file transfer operation for the one or more of the plurality of files from the identified one or more file offsets;
to determine whether the one or more of the plurality of files meets a resume criteria prior to the resumption of the file transfer operation for the one or more of the plurality of files;
wherein, in determining whether the one or more of the plurality of files meets the resume criteria, said at least one processing platform is configured:
to determine whether the one or more of the plurality of files have been deleted from the source storage device prior to the resumption of the file transfer operation for the one or more of the plurality of files and are unavailable for transmission to the cloud storage platform during a resumed file transfer operation for the one or more of the plurality of files; and
to determine whether the one or more of the plurality of files on the source storage device have been modified prior to the resumption of the file transfer operation for the one or more of the plurality of files;
to cause deletion from the one or more databases of information corresponding to the one or more of the plurality of files that have been deleted from the source storage device;
to cause omission of the one or more of the plurality of files that have been modified from the resumed file transfer operation;
to cause archiving to the cloud storage platform of the one or more of the plurality of files that have been modified in another file transfer operation; and
to cause deletion from the cloud storage platform of one or more partial objects corresponding to the one or more of the plurality of files that have been modified, wherein the one or more partial objects were partially written to the cloud storage platform due to the failure of the file transfer operation.
2. The apparatus of claim 1 wherein, in determining whether the one or more of the plurality of files meets the resume criteria, said at least one processing platform is further configured to determine whether the one or more of the plurality of files has exceeded a threshold for a number of times to resume a failed file transfer operation.
3. The apparatus of claim 1 wherein said at least one processing platform is further configured:
to cause deletion of one or more objects from the cloud storage platform corresponding to the one or more of the plurality of files that have been deleted from the source storage device; and
to cause omission of the one or more of the plurality of files that have been deleted from the source storage device from the resumed file transfer operation.
4. The apparatus of claim 1 wherein, in determining whether the one or more of the plurality of files have been modified, said at least one processing platform is configured to compare one or more timestamps of the one or more of the plurality of files on the source storage device with one or more timestamps of the one or more of the plurality of files stored in the one or more databases.
5. The apparatus of claim 1 wherein, in determining whether the one or more of the plurality of files meets the resume criteria, said at least one processing platform is further configured to determine whether the one or more of the plurality of files has exceeded a file size threshold.
6. The apparatus of claim 1 wherein said at least one processing platform is further configured to cause modification of a size of cloud objects for the one or more of the plurality of files to correspond to the identified one or more file offsets.
7. The apparatus of claim 1 wherein said at least one processing platform is further configured to remove from the one or more databases the information corresponding to the one or more of the plurality of files following successful completion of the file transfer operation.
8. The apparatus of claim 1 wherein the information corresponding to the plurality of files comprises for each of the plurality of files at least one of an access timestamp, a modified timestamp, a changed timestamp, a file size, a file path, an object identifier, a read offset, a storage protocol and a number of times to a failed file transfer operation was resumed.
9. The apparatus of claim 1 wherein said at least one processing platform comprises a cloud tiering appliance.
10. A method comprising:
storing information corresponding to a plurality of files being written from a source storage device to a cloud storage platform in a file transfer operation, wherein the information corresponding to the plurality of files is stored in one or more databases;
in response to a failure of the file transfer operation, identifying from the information one or more file offsets successfully written to the cloud storage platform for one or more of the plurality of the files prior to the failure;
causing resumption of the file transfer operation for the one or more of the plurality of files from the identified one or more file offsets;
determining whether the one or more of the plurality of files meets a resume criteria prior to the resumption of the file transfer operation for the one or more of the plurality of files;
wherein determining whether the one or more of the plurality of files meets the resume criteria comprises:
determining whether the one or more of the plurality of files have been deleted from the source storage device prior to the resumption of the file transfer operation for the one or more of the plurality of files and are unavailable for transmission to the cloud storage platform during a resumed file transfer operation for the one or more of the plurality of files; and
determining whether the one or more of the plurality of files on the source storage device have been modified prior to the resumption of the file transfer operation for the one or more of the plurality of files;
causing deletion from the one or more databases of information corresponding to the one or more of the plurality of files that have been deleted from the source storage device;
causing omission of the one or more of the plurality of files that have been modified from the resumed file transfer operation;
cause archiving to the cloud storage platform of the one or more of the plurality of files that have been modified in another file transfer operation; and
causing deletion from the cloud storage platform of one or more partial objects corresponding to the one or more of the plurality of files that have been modified, wherein the one or more partial objects were partially written to the cloud storage platform due to the failure of the file transfer operation
wherein the method is performed by at least one processing platform comprising at least one processing device comprising a processor coupled to a memory.
11. The method of claim 10 further comprising:
causing deletion of one or more objects from the cloud storage platform corresponding to the one or more of the plurality of files that have been deleted from the source storage device; and
causing omission of the one or more of the plurality of files that have been deleted from the source storage device from the resumed file transfer operation.
12. A computer program product comprising a non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code when executed by at least one processing platform causes said at least one processing platform:
to store information corresponding to a plurality of files being written from a source storage device to a cloud storage platform in a file transfer operation, wherein the information corresponding to the plurality of files is stored in one or more databases;
in response to a failure of the file transfer operation, to identify from the information one or more file offsets successfully written to the cloud storage platform for one or more of the plurality of the files prior to the failure;
to cause resumption of the file transfer operation for the one or more of the plurality of files from the identified one or more file offsets;
to determine whether the one or more of the plurality of files meets a resume criteria prior to the resumption of the file transfer operation for the one or more of the plurality of files;
wherein, in determining whether the one or more of the plurality of files meets the resume criteria, the program code causes said at least one processing platform:
to determine whether the one or more of the plurality of files have been deleted from the source storage device prior to the resumption of the file transfer operation for the one or more of the plurality of files and are unavailable for transmission to the cloud storage platform during a resumed file transfer operation for the one or more of the plurality of files; and
to determine whether the one or more of the plurality of files on the source storage device have been modified prior to the resumption of the file transfer operation for the one or more of the plurality of files;
to cause deletion from the one or more databases of information corresponding to the one or more of the plurality of files that have been deleted from the source storage device;
to cause omission of the one or more of the plurality of files that have been modified from the resumed file transfer operation;
to cause archiving to the cloud storage platform of the one or more of the plurality of files that have been modified in another file transfer operation; and
to cause deletion from the cloud storage platform of one or more partial objects corresponding to the one or more of the plurality of files that have been modified, wherein the one or more partial objects were partially written to the cloud storage platform due to the failure of the file transfer operation.
13. The computer program product of claim 12 wherein, in determining whether the one or more of the plurality of files meets the resume criteria, the program code further causes said at least one processing platform to determine whether the one or more of the plurality of files has exceeded a threshold for a number of times to resume a failed file transfer operation.
14. The computer program product of claim 12 wherein the program code further causes said at least one processing platform:
to cause deletion of one or more objects from the cloud storage platform corresponding to the one or more of the plurality of files that have been deleted from the source storage device; and
to cause omission of the one or more of the plurality of files that have been deleted from the source storage device from the resumed file transfer operation.
15. The computer program product of claim 12 wherein, in determining whether the one or more of the plurality of files have been modified, the program code causes said at least one processing platform to compare one or more timestamps of the one or more of the plurality of files on the source storage device with one or more timestamps of the one or more of the plurality of files stored in the one or more databases.
16. The computer program product of claim 12 wherein, in determining whether the one or more of the plurality of files meets the resume criteria, the program code further causes said at least one processing platform to determine whether the one or more of the plurality of files has exceeded a file size threshold.
17. The computer program product of claim 12 wherein the program code further causes said at least one processing platform to cause modification of a size of cloud objects for the one or more of the plurality of files to correspond to the identified one or more file offsets.
18. The computer program product of claim 12 wherein the program code further causes said at least one processing platform to remove from the one or more databases the information corresponding to the one or more of the plurality of files following successful completion of the file transfer operation.
19. The method of claim 10 wherein determining whether the one or more of the plurality of files have been modified comprises comparing one or more timestamps of the one or more of the plurality of files on the source storage device with one or more timestamps of the one or more of the plurality of files stored in the one or more databases.
20. The method of claim 10 wherein determining whether the one or more of the plurality of files meets the resume criteria further comprises determining whether the one or more of the plurality of files has exceeded a file size threshold.