Metadata journal in a distributed storage system
A plurality of computing devices are communicatively coupled to each other via a network, and each of the plurality of computing devices is operably coupled to one or more of a plurality of storage devices. Each computing device is operable to compress one or more blocks of data and append a journal in front of the data. The journal and the data are written concurrently to flash memory. Each computing device is also operable to maintain a metadata registry that records changes in the flash memory. In the event of a power failure, the journal and previous journals may be used to verify the state of the metadata registry.
1. A system comprising:
a storage device; and
a plurality of computing devices communicatively coupled to each other and to the storage device via a network, wherein:
each of the plurality of computing devices is operable to write one or more blocks of data to the storage device,
metadata is provided to a metadata journal as the one or more blocks of data are compressed,
the metadata journal is associated with the changes being made to the storage device by the one or more blocks of data,
the metadata journal comprises one or more back pointers to one or more previous metadata journals in the storage device,
the metadata journal and the one or more previous metadata journals form a linked list that keeps track of higher level operations and ensures a persistence of the higher level operations in case of a failure,
the metadata journal and the one or more previous metadata journals are configured to be replayed either sequentially or in parallel, and
the system is operable to determine between a sequential replay and a parallel replay at the time of a power failure.
2. The system of claim 1 , wherein each of the plurality of computing devices comprises a virtual file system (VFS) node.
3. The system of claim 2 , wherein a metadata registry is maintained on a VFS backend in the VFS node.
4. The system of claim 3 , wherein the VFS backend manages the metadata registry to store the changes being made to the storage device by the one or more blocks of data.
5. The system of claim 4 , wherein the metadata registry comprises volatile memory on the VFS backend.
6. The system of claim 1 , wherein the storage device comprises flash memory.
7. The system of claim 1 , wherein the metadata journal is written to the storage device concurrently with the compressed one or more blocks of data.
8. The system of claim 7 , wherein in the event of a power failure, the metadata journal and the one or more previous metadata journals are compared to data that is saved in the storage device to determine whether the saved data is intact.
9. The system of claim 8 , wherein if the saved data is intact, the one or more previous metadata journals are discarded.
10. The system of claim 8 , wherein if the saved data is not intact, the metadata journal and the one or more previous metadata journals are replayed to determine a new metadata registry state.
11. A method comprising:
compressing one or more blocks of data via a computing device, wherein the compressing provides, to a metadata journal, metadata associated with changes made to a storage device by the one or more blocks of data;
appending the metadata journal to the compressed one or more blocks of data; and
concurrently writing the metadata journal and the compressed one or more blocks of data to the storage device, wherein:
the metadata journal comprises the metadata associated with changes made to the storage device by the one or more blocks of data,
the metadata journal comprises one or more back pointers to one or more previous metadata journals in the storage device,
the metadata journal and the one or more previous metadata journals form a linked list that keeps track of higher level operations and ensures a persistence of the higher level operations in case of a failure,
the metadata journal and the one or more previous metadata journals are configured to be replayed either sequentially or in parallel, and
determining between a sequential replay and a parallel replay occurs at the time of a power failure.
12. The method of claim 11 , wherein the computing device comprises a virtual file system (VFS) node.
13. The method of claim 12 , wherein the method comprises maintaining a metadata registry related to the metadata journal on a VFS backend in the VFS node.
14. The method of claim 13 , wherein the method comprises managing, via the VFS backend, the metadata registry to store the changes being made to the storage device by the one or more blocks of data.
15. The method of claim 14 , wherein the metadata registry comprises volatile memory on the VFS backend.
16. The method of claim 11 , wherein the storage device comprises flash memory.
17. The method of claim 11 , wherein the method comprises comparing the metadata journal and the one or more previous metadata journals to data that is saved on the storage device, in the event of a power failure, to determine whether the saved data is intact.
18. The method of claim 17 , wherein the method comprises discarding the one or more previous metadata journals if the saved data is intact.
19. The method of claim 17 , wherein the method comprises replaying the metadata journal and the one or more previous metadata journals, if the saved data is not intact, to determine a new metadata registry state.