Two level addressing in storage clusters
Digital objects are stored and accessed within a fixed content storage cluster by using a page mapping table and a pages index. A stream is read from the cluster by using a portion of its unique identifier as a key into the page mapping table. The page mapping table indicates a node holding a pages index indicating where the stream is stored. A stream is written by storing the stream on any suitable node and then updating a pages index stored within the cluster responsible for knowing the location of digital objects having unique identifiers that fall within a particular address range. The cluster recovers from a node failure by first replicating streams from the failed node and reallocating a page mapping table to create a new pages index. The remaining nodes send records of the unique identifiers corresponding to objects they hold to the new pages index.
1. A method of reading a digital object from a fixed-content storage cluster, said method comprising:
receiving a unique identifier at a first computer node within a storage cluster having a plurality of computer nodes, said storage cluster including stored digital objects in persistent storage, each of said digital objects being identified by a unique identifier and each of said unique identifiers only identifying a single digital object in said fixed-content storage cluster;
identifying a first portion of said unique identifier, said first portion having a value within a range of all possible values;
mapping said value indicated by said first portion of said unique identifier to identify a second computer node;
accessing a second disk index in RAM on said second computer node using said unique identifier to identify a third computer node where said digital object is stored, said second disk index including unique identifiers of digital objects stored on said second computer node and locations on disk of said digital objects;
accessing a third disk index in RAM on said third computer node using said unique identifier to identify where on disk of said third computer node said digital object is stored, said third disk index including unique identifiers of digital objects stored on said third computer node and locations on disk of said digital objects;
and reading said digital object identified by said unique identifier from said third computer node and returning said digital object to a software application.
2. The method as recited in claim 1 wherein said storage cluster is a write-once read-many (WORM) fixed-content storage cluster.
3. The method as recited in claim 1 wherein:
accessing said second disk index on said second computer node using said unique identifier identifies all computer nodes within said storage cluster where said digital object is stored.
4. The method as recited in claim 1 wherein said steps of claim 1 other than said reading step do not require a disk access.
5. The method as recited in claim 1 wherein accessing said second disk index further includes:
mapping said unique identifier to a volume identifier identifying a disk within said storage cluster; and
mapping said volume identifier to said third computer node.
6. The method as recited in claim 1 wherein said accessing a third disk index in RAM on said third computer node further includes:
identifying a volume identifier of said disk where said digital object is stored; and
identifying a location on said disk where said digital object is stored.
7. The method as recited in claim 1 wherein said first, second and third computer nodes are the same node.