Page-based metadata system for distributed filesystem
Methods and systems are described for storing metadata in a network of distributed remote storage devices. An example computer implemented method includes identifying a plurality of metadata inodes, grouping the plurality of metadata inodes into one or more metadata pages, mapping the plurality of metadata inodes and the plurality of metadata pages, and storing the mapping as a manifest page, and storing data to and retrieving data from the distributed remote storage devices using the plurality of metadata inodes.
1. A computer implemented method for retrieving metadata from a network of distributed remote storage devices, comprising:
referencing a metadata manifest page for a metadata page location of at least one metadata inode within a metadata page based at least in part on an identifier mapping stored at the metadata manifest page, wherein the metadata manifest page is different from the metadata page;
obtaining information associated with the metadata page location of the at least one metadata inode based at least in part on an inode identifier and a page identifier associated with the at least one metadata inode, wherein the identifier mapping comprises a mapping from the inode identifier to the page identifier; and
accessing the at least one metadata inode based at least in part on the obtained information.
2. The method of claim 1 , further comprising:
identifying a plurality of metadata inodes, wherein the at least one metadata inode is included in the plurality of metadata inodes; and
grouping the plurality of metadata inodes into one or more metadata pages based at least in part on at least one criteria.
3. The method of claim 2 , wherein the at least one criteria include a type of metadata inode.
4. The method of claim 2 , further comprising:
mapping the plurality of metadata inodes and the one or more metadata pages, wherein the metadata manifest page is based at least in part on the mapping.
5. The method of claim 4 , further comprising:
storing the mapping as the metadata manifest page; and
storing data to the distributed remote storage devices using the plurality of metadata inodes.
6. The method of claim 2 , further comprising:
conducting a search by first obtaining at least one metadata page associated with the at least one metadata inode followed by obtaining the at least one metadata inode, wherein accessing the at least one metadata inode is based at least in part on the conducted search.
7. The method of claim 2 , wherein the plurality of metadata inodes include inode identifiers, and the one or more metadata pages include page identifiers.
8. The method of claim 7 , further comprising:
mapping the plurality of metadata inodes and the one or more metadata pages based at least in part on the inode identifiers and the page identifiers, wherein the metadata manifest page is based at least in part on the mapping.
9. The method of claim 1 , wherein the at least one metadata inode includes at least one of a directory inode, a file inode, and a symlink inode.
10. The method of claim 1 , further comprising:
creating additional metadata pages when additional distributed remote storage devices are added to the network; and
storing the additional metadata pages on an existing metadata page.
11. The method of claim 10 , further comprising:
creating additional metadata manifest pages with one or more identifiers when additional distributed remote storage devices are added to the network, wherein the one or more identifiers are associated with the additional metadata pages.
12. An apparatus for retrieving metadata from a network of distributed remote storage devices, comprising:
a processor;
a memory in electronic communication with the processor; and
instructions stored in the memory, the instructions being executable by the processor to:
reference a metadata manifest page for a metadata page location of at least one metadata inode within a metadata page based at least in part on an identifier mapping stored at the metadata manifest page, wherein the metadata manifest page is different from the metadata page;
obtain information associated with the metadata page location of the at least one metadata inode based at least in part on an inode identifier and a page identifier associated with the at least one metadata inode, wherein the identifier mapping comprises a mapping from the inode identifier to the page identifier; and
access the at least one metadata inode based at least in part on the obtained information.
13. The apparatus of claim 12 , wherein the instructions are executable by the processor to:
identify a plurality of metadata inodes, wherein the at least one metadata inode is included in the plurality of metadata inodes; and
group the plurality of metadata inodes into one or more metadata pages based at least in part on at least one criteria.
14. The apparatus of claim 13 , wherein the at least one criteria include a type of metadata inode.
15. The apparatus of claim 13 , wherein the instructions are executable by the processor to:
map the plurality of metadata inodes and the one or more metadata pages, wherein the metadata manifest page is based at least in part on the mapping.
16. The apparatus of claim 15 , wherein the instructions are executable by the processor to:
store the mapping as the metadata manifest page; and
store data to the distributed remote storage devices using the plurality of metadata inodes.
17. The apparatus of claim 13 , wherein the instructions are executable by the processor to:
conduct a search by first obtaining at least one metadata page associated with the at least one metadata inode followed by obtaining the at least one metadata inode, wherein accessing the at least one metadata inode is based at least in part on the conducted search.
18. A computer-program product for storing metadata in a network of distributed remote storage devices, the computer-program product comprising a non-transitory computer-readable medium storing instructions executable by a processor to:
reference a metadata manifest page for a metadata page location of at least one metadata inode within a metadata page based at least in part on an identifier mapping stored at the metadata manifest page, wherein the metadata manifest page is different from the metadata page;
obtain information associated with the metadata page location of the at least one metadata inode based at least in part on an inode identifier and a page identifier associated with the at least one metadata inode, wherein the identifier mapping comprises a mapping from the inode identifier to the page identifier; and
access the at least one metadata inode based at least in part on the obtained information.
19. The computer-program product of claim 18 , wherein the instructions are executable by the processor to:
identify a plurality of metadata inodes, wherein the at least one metadata inode is included in the plurality of metadata inodes; and
group the plurality of metadata inodes into one or more metadata pages based at least in part on at least one criteria.
20. The computer-program product of claim 19 , wherein the at least one criteria include a type of metadata inode.