System for transition from a hierarchical file system to an object store
Techniques are provided for transitioning from a hierarchical file system to an object store. A request is received to change file metadata of a hierarchical file system. In response to the request: in-cache file metadata of the hierarchical file system is modified. The in-cache file metadata includes a directory structure of the hierarchical file system and file metadata of the hierarchical file system. Additionally, an in-store metadata journal entry is generated. The in-store metadata journal entry indicates an update to in-store file metadata. The in-store file metadata includes the directory structure of the hierarchical file system and file metadata of the hierarchical file system and is stored in an object store. The in-store metadata journal entry is stored in the object store. Furthermore, in-store file metadata and in-store metadata journal entry may be used to reconstruct the in-cache file metadata on another node.
1. A method comprising:
storing in-store file metadata as one or more replace-only objects in an object store, the in-store file metadata including a directory structure of a hierarchical file system and file metadata of the hierarchical file system;
wherein updating the in-store file metadata requires a full re-write of the in-store file metadata;
receiving a request to change the file metadata of the hierarchical file system; and
in response to the request:
modifying in-cache file metadata of the hierarchical file system, wherein the in-cache file metadata includes the directory structure of the hierarchical file system and the file metadata of the hierarchical file system;
generating an in-store metadata journal entry, wherein the in-store metadata journal entry indicates an update to the in-store file metadata; and
storing the in-store metadata journal entry as one or more replace-only objects in the object store.
2. The method of claim 1 , wherein the in-cache file metadata does not store file data of the hierarchical file system.
3. The method of claim 1 , wherein the in-cache file metadata stores a globally unique identifier (GUID) for each file of the hierarchical file system that has successfully been copied to the object store.
4. The method of claim 1 , wherein the object store stores a GUID for each object in the object store.
5. The method of claim 1 , wherein the in-cache file metadata is stored in-memory or on disk on a client-side server.
6. The method of claim 1 , wherein the hierarchical file system is stored in-memory on a client-side server.
7. The method of claim 1 , further comprising;
receiving an I/O request for a file;
in response to the request, determining that the in-cache file metadata includes a GUID that corresponds to the file; and
directing the request to the object store.
8. The method of claim 1 , further comprising;
receiving an I/O request for a file;
in response to the request, determining that the in-cache file metadata does not include a GUID that corresponds to the file; and
directing the request to the hierarchical file system.
9. The method of claim 1 , further comprising;
receiving a request to access file metadata of the hierarchical file system; and
responding to the request by accessing the in-cache file metadata.
10. The method of claim 1 , further comprising;
receiving a request to move or rename a file of the hierarchical file system;
in response to the request:
modifying the in-cache file metadata of the hierarchical file system;
generating a particular in-store metadata journal entry; and
storing the particular in-store metadata journal entry in the object store.
11. The method of claim 1 , wherein the in-cache file metadata is stored on a first host node, further comprising:
reconstructing the in-cache file metadata on a second host node based on the in-store file metadata and the in-store metadata journal entry.
12. A non-transitory computer-readable storage medium storing sequences of instructions that, when executed by one or more processors, cause:
storing in-store file metadata as one or more replace-only objects in an object store, the in-store file metadata including a directory structure of a hierarchical file system and file metadata of the hierarchical file system;
wherein updating the in-store file metadata requires a full re-write of the in-store file metadata;
receiving a request to change the file metadata of the hierarchical file system; and
in response to the request:
modifying in-cache file metadata of the hierarchical file system, wherein the in-cache file metadata includes the directory structure of the hierarchical file system and the file metadata of the hierarchical file system;
generating an in-store metadata journal entry, wherein the in-store metadata journal entry indicates an update to the in-store file metadata; and
storing the in-store metadata journal entry as one or more replace-only objects in the object store.
13. The non-transitory computer-readable storage medium of claim 12 , wherein the in-cache file metadata does not store file data of the hierarchical file system.
14. The non-transitory computer-readable storage medium of claim 12 , wherein the in-cache file metadata stores a globally unique identifier (GUID) for each file of the hierarchical file system that has successfully been copied to the object store.
15. The non-transitory computer-readable storage medium of claim 12 , wherein the object store stores a GUID for each object in the object store.
16. The non-transitory computer-readable storage medium of claim 12 , wherein the in-cache file metadata is stored in-memory or on disk on a client-side server.
17. The non-transitory computer-readable storage medium of claim 12 , wherein the hierarchical file system is stored in-memory on a client-side server.
18. The non-transitory computer-readable storage medium of claim 12 , the sequences of instructions including instructions that, when executed by said one or more processors, cause:
receiving an I/O request for a file;
in response to the request, determining that the in-cache file metadata includes a GUID that corresponds to the file;
directing the request to the object store.
19. The non-transitory computer-readable storage medium of claim 12 , the sequences of instructions including instructions that, when executed by said one or more processors, cause:
receiving an I/O request for a file;
in response to the request, determining that the in-cache file metadata does not include a GUID that corresponds to the file;
directing the request to the hierarchical file system.
20. The non-transitory computer-readable storage medium of claim 12 , the sequences of instructions including instructions that, when executed by said one or more processors, cause:
receiving a request to access file metadata of the hierarchical file system;
responding to the request by accessing the in-cache file metadata.
21. The non-transitory computer-readable storage medium of claim 12 , the sequences of instructions including instructions that, when executed by said one or more processors, cause: receiving a request to move or rename a file of the hierarchical file system;
in response to the request:
modifying the in-cache file metadata of the hierarchical file system;
generating a particular in-store metadata journal entry;
storing the particular in-store metadata journal entry in the object store.
22. The non-transitory computer-readable storage medium of claim 12 , wherein the in-cache file metadata is stored on a first host node; the sequences of instructions including instructions that, when executed by said one or more processors, cause:
reconstructing the in-cache file metadata on a second host node based on the in-store file metadata and the in-store metadata journal entry.