Page-based metadata system for distributed filesystem
View Patent ↗Methods and systems are described for storing metadata in a network of distributed remote storage devices. An example computer implemented method includes identifying a plurality of metadata inodes, grouping the plurality of metadata inodes into one or more metadata pages, mapping the plurality of metadata inodes and the plurality of metadata pages, and storing the mapping as a manifest page, and storing data to and retrieving data from the distributed remote storage devices using the plurality of metadata inodes.
1. A computer implemented method for storing metadata in a network of distributed remote storage devices, comprising:
identifying a plurality of metadata inodes and assigning an associated inode identifier to each of the plurality of metadata inodes;
grouping the plurality of metadata inodes into one or more metadata pages and assigning an associated page identifier to each of the one or more metadata pages;
mapping the plurality of metadata inodes and the one or more metadata pages, and storing the mapping as a manifest page, wherein the manifest page comprises a mapping from each of the plurality of inode identifiers to each of the one or more page identifiers; and
storing data to the distributed remote storage devices using the plurality of metadata inodes; and
retrieving data from the distributed remote storage devices using a lookup function associated with the plurality of metadata inodes, the lookup function comprising referencing the manifest page, obtaining information associated with the manifest page and the plurality of metadata inodes based at least in part on an inode identifier and a page identifier, and conducting a search by obtaining the metadata page and the metadata inode.
2. The method of claim 1 , wherein retrieving data from the distributed remote storage devices includes referencing the manifest page for a location of at least one of the metadata inodes, and then accessing the at least one of the metadata inodes based on the location.
3. The method of claim 1 , wherein the metadata inodes include at least one of a directory inode, a file inode, and a symlink inode.
4. The method of claim 1 , further comprising:
creating additional metadata pages when additional distributed remote storage devices are added to the network.
5. The method of claim 1 , further comprising:
revising or removing one or more of the metadata pages when at least one of the distributed remote storage devices is removed from the network.
6. The method of claim 1 , further comprising:
storing the metadata pages and the manifest page in a filesystem for the network.
7. The method of claim 1 , wherein grouping the plurality of metadata inodes is based on at least one criteria.
8. The method of claim 7 , wherein the at least one criteria includes a type of metadata inode.
9. The method of claim 1 , wherein grouping the plurality of metadata inodes into one or more metadata pages further comprises:
grouping two or more of the plurality of metadata inodes into a single metadata page.
10. A computer-program product for storing metadata in a network of distributed remote storage devices, the computer-program product comprising a non-transitory computer-readable medium storing instructions executable by a processor to:
identify a plurality of metadata inodes and assign an associated inode identifier to each of the plurality of metadata inodes;
group the plurality of metadata inodes into at least one metadata page and assign an associated page identifier to each of the at least one metadata page;
map the plurality of metadata inodes and the at least one at least one metadata page;
store the map as a manifest page, wherein the manifest page comprises a mapping from each of the plurality of inode identifiers to each of the at least one page identifiers;
store the at least one metadata page and the manifest page in a filesystem for the network; and
retrieve data from the distributed remote storage devices using a lookup function associated with the plurality of metadata inodes, the lookup function comprising referencing the manifest page, obtaining information associated with the manifest page and the plurality of metadata inodes based at least in part on an inode identifier and a page identifier, and conducting a search by obtaining the metadata page and the metadata inode.
11. The computer-program product of claim 10 , wherein the instructions are executable by the processor to:
receive additional metadata inodes for at least one additional distributed remote storage device added to the network; and
add the additional metadata inodes to one of the at least one metadata page or create at least one additional metadata page for the additional metadata inodes.
12. The computer-program product of claim 11 , wherein the instructions are executable by the processor to:
add data to the distributed remote storage devices using the plurality of additional metadata inodes.
13. The computer-program product of claim 11 , wherein the instructions are executable by the processor to:
update the manifest page to include mapping of the additional metadata inodes.
14. The computer-program product of claim 10 , wherein the instructions are executable by the processor to:
store multiple copies of the metadata inodes in the filesystem.
15. The computer-program product of claim 10 , wherein the instructions are executable by the processor to:
group two or more of the plurality of metadata inodes into a single metadata page.
16. An apparatus for storing metadata in a network of distributed remote storage devices, comprising:
a processor;
a memory in electronic communication with the processor; and
instructions stored in the memory, the instructions being executable by the processor to:
identify a plurality of metadata inodes and assign an associated inode identifier to each of the plurality of metadata inodes;
group the plurality of metadata inodes into at least one metadata page and assign an associated page identifier to each of the at least one metadata page;
map the plurality of metadata inodes and the at least one metadata page;
store the map as a manifest page, wherein the manifest page comprises a mapping from each of the plurality of inode identifiers to each of the at least one page identifiers;
store data to the distributed remote storage devices using the plurality of metadata inodes; and
retrieve data from the distributed remote storage devices using a lookup function associated with the plurality of metadata inodes, the lookup function comprising referencing the manifest page, obtaining information associated with the manifest page and the plurality of metadata inodes based at least in part on an inode identifier and a page identifier, and conducting a search by obtaining the metadata page and the metadata inode.
17. The apparatus of claim 16 , wherein the instructions are executable by the processor to:
store the at least one metadata page and the manifest page in a filesystem for the network.
18. The apparatus of claim 16 , wherein the instructions are executable by the processor to:
divide the manifest page into a plurality of manifest pages.
19. The apparatus of claim 16 , wherein the metadata includes at least one structure data with information about other inode objects, names of the inode objects, and cryptographic information.
20. The apparatus of claim 16 , wherein the instructions are executable by the processor to:
group two or more of the plurality of metadata inodes into a single metadata page.