Supporting multiple operations in transaction logging for a cloud-enabled file system
Methods, apparatus, and processor-readable storage media for supporting multiple operations in transaction logging for a cloud enabled file system are provided herein. An example computer-implemented method includes obtaining a plurality of file system operations to be performed on a cloud enabled file system; executing the plurality of file system operations as a single file system transaction; and maintaining a transaction log for the single transaction, the transaction log comprising information for one or more sub-transactions that were completed in conjunction with said executing, wherein the one or more sub-transactions correspond to at least a portion of the plurality of file system operations.
1. A computer-implemented method comprising: obtaining a sequence of file system operations to be performed on a cloud-enabled file system, wherein the cloud-enabled file system is maintained on a local storage array and comprises at least one stub file that is indicative of a location of an object in a cloud storage system, the object comprising data related to a file previously sent from the local storage array to the cloud storage system;
obtaining a logging descriptor fora first file system operation in the sequence of file system operations;
executing the sequence of file system operations as a single file system transaction;
maintaining a transaction log for the single file system transition, the transaction log comprising information for a plurality of sub-transactions that were completed in conjunction with said executing, wherein the plurality of sub-transactions are associated with at least the first file system operation and a second file system operation that is subsequent to the first file system operation in the sequence of file system operations, wherein at least one of the first and second file system operations comprises a sequence of multiple sub-transactions performed on metadata corresponding to the at least one stub file, and wherein said maintaining comprises updating the logging descriptor obtained for the first file system operation based on a number of sub-transactions corresponding to the second file system operation; and
sending information to the cloud-enabled file system indicating a completion of the sequence of file system operations based at least in part on the updated logging descriptor;
wherein the method is performed by at least one processing device comprising a processor coupled to a memory.
2. The computer-implemented method of claim 1 , further comprising: in response to one or more of a system reboot and a system recovery, determining that at least one sub-transaction corresponding to at least one of the file system operations in the sequence of file system operations was not completed; and discarding the single file system transaction based on a result of the determining.
3. The computer-implemented method of claim 1 , wherein the sending the information comprises issuing a commit of the updated logging descriptor to said cloud-enabled file system.
4. The computer-implemented method of claim 1 , further comprising: maintaining, by the local storage array, a first database comprising hard links corresponding to the at least one stub file; and maintaining a second database comprising at least one hard link corresponding to at least one further stub file, wherein the at least one further stub file was one of: previously deleted from said cloud-enabled file system, and previously rehydrated with data from a corresponding object in the cloud storage system.
5. The computer-implemented method of claim 1 , wherein the sequence of file system operations corresponds to one of: deleting a stub file from the cloud-enabled file system; rehydrating a stub file in the cloud-enabled file system with data from an object stored in the cloud storage system; and executing an orphan management process to remove orphaned files in the cloud-enabled file system.
6. The computer-implemented method of claim 1 , wherein the cloud-enabled file system comprises a UFS64 file system.
7. The computer-implemented method of claim 1 , wherein the sequence of multiple sub-transactions performed on the metadata corresponding to the at least one stub file comprises a first sub-transaction with the local storage array and a second sub-transaction with the cloud storage system.
8. A non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code when executed by at least one processing device causes the at least one processing device:
to obtain a sequence of file system operations to be performed on a cloud-enabled file system, wherein the cloud-enabled file system is maintained on a local storage array and comprises at least one stub file that is indicative of a location of an object in a cloud storage system, the object comprising data related to a file previously sent from the local storage array to the cloud storage system;
to obtain a logging descriptor for a first file system operation in the sequence of file system operations;
to execute the sequence of file system operations as a single file system transaction;
to maintain a transaction log for the single file system transaction, the transaction log comprising information for a plurality of sub-transactions that were completed in conjunction with said executing, wherein the plurality of sub-transactions are associated with at least the first file system operation and a second file system operation that is subsequent to the first file system operation in the sequence of file system operations, wherein at least one of the first and second file system operations comprises a sequence of multiple sub-transactions performed on metadata corresponding to the at least one stub file, and wherein said maintaining comprises updating the logging descriptor obtained for the first file system operation based on a number of sub-transactions corresponding to the second file system operation; and
to send information to the cloud-enabled file system indicating a completion of the sequence of file system operations based at least in part on the updated logging descriptor.
9. The non-transitory processor-readable storage medium of claim 8 , wherein the at least one processing device is further caused: to determine, in response to one or more of a system reboot and a system recovery, that at least one sub-transaction corresponding to at least one of the file system operations in the sequence of file system operations was not completed; and to discard the single file system transaction based on a result of the determining.
10. The non-transitory processor-readable storage medium of claim 8 , wherein the sending the information comprises issuing a commit of the updated logging descriptor to said cloud-enabled file system.
11. The non-transitory processor-readable storage medium of claim 8 , wherein the at least one processing device is further caused:
maintain, by the local storage array, a first database comprising hard links corresponding to the at least one stub file; and
maintain a second database comprising at least one hard link corresponding to at least one further stub file, wherein the at least one further stub file was one of: previously deleted from said cloud-enabled file system, and previously rehydrated with data from a corresponding object in the cloud storage system.
12. The non-transitory processor-readable storage medium of claim 8 , wherein the sequence of file system operations corresponds to one of:
deleting a stub file from the cloud-enabled file system;
rehydrating a stub file in the cloud-enabled file system with data from an object stored in the cloud storage system; and
executing an orphan management process to remove orphaned files in the cloud-enabled file system.
13. The non-transitory processor-readable storage medium of claim 8 , wherein the sequence of multiple sub-transactions performed on the metadata corresponding to the at least one stub file comprises a first sub-transaction with the local storage array and a second sub-transaction with the cloud storage system.
14. An apparatus comprising: at least one processing device comprising a processor coupled to a memory; the at least one processing device being configured:
to obtain a sequence of file system operations to be performed on a cloud-enabled file system, wherein the cloud-enabled file system is maintained on a local storage array and comprises at least one stub file that is indicative of a location of an object in a cloud storage system, the object comprising data related to a file previously sent from the local storage array to the cloud storage system;
to obtain a logging descriptor for a first file system operation in the sequence of file system operations;
to execute the sequence of file system operations as a single file system transaction;
to maintain a transaction log for the single file system transaction, the transaction log comprising information for a plurality of sub-transactions that were completed in conjunction with said executing, wherein the plurality of sub-transactions are associated with at least the first file system operation and a second file system operation that is subsequent to the first file system operation in the sequence of file system operations, wherein at least one of the first and second file system operations comprises a sequence of multiple sub-transactions performed on metadata corresponding to the at least one stub file, and wherein said maintaining comprises updating the logging descriptor obtained for the first file system operation based on a number of sub-transactions corresponding to the second file system operation; and
to send information to the cloud-enabled file system indicating a completion of the sequence of file system operations based at least in part on the updated logging descriptor.
15. The apparatus of claim 14 , wherein the at least one processing device is further configured: to determine, in response to one or more of a system reboot and a system recovery, that at least one sub-transaction corresponding to at least one of the file system operations in the sequence of file system operations was not completed; and to discard the single file system transaction based on a result of the determining.
16. The apparatus of claim 14 , wherein the sending the information comprises issuing a commit of the updated logging descriptor to said cloud-enabled file system.
17. The apparatus of claim 14 , wherein the sequence of multiple sub-transactions performed on the metadata corresponding to the at least one stub file comprises a first sub-transaction with the local storage array and a second sub-transaction with the cloud storage system.
18. The apparatus of claim 14 , wherein the at least one processing device is further configured: to maintain, by the local storage array, a first database comprising hard links corresponding to the at least one stub file; and to maintain a second database comprising at least one hard link corresponding to at least one further stub file, wherein the at least one further stub file was one of: previously deleted from said cloud-enabled file system, and previously rehydrated with data from a corresponding object in the cloud storage system.
19. The apparatus of claim 14 , wherein the sequence of file system operations corresponds to one of: deleting a stub file from the cloud-enabled file system; rehydrating a stub file in the cloud-enabled file system with data from an object stored in the cloud storage system; and executing an orphan management process to remove orphaned files in the cloud-enabled file system.
20. The apparatus of claim 14 , wherein the cloud-enabled file system comprises a UFS64 file system.