Two level addressing in storage clusters
Digital objects are stored and accessed within a fixed content storage cluster by using a page mapping table and a pages index. A stream is read from the cluster by using a portion of its unique identifier as a key into the page mapping table. The page mapping table indicates a node holding a pages index indicating where the stream is stored. A stream is written to the cluster by storing the stream on any suitable node and then updating a pages index stored within the cluster. The cluster recovers from a node failure by first replicating streams from the failed node and reallocating a page mapping table to create a new pages index. The remaining nodes send records of the unique identifiers corresponding to objects they hold to the new pages index. A node is added to the cluster by reallocating a page mapping table.
1. A method of reading a digital object from a fixed-content storage cluster, said method comprising:
receiving from a software application a unique identifier at a first computer node within a storage cluster having a plurality of computer nodes, said storage cluster including stored digital objects in persistent storage, each of said digital objects being identified by a unique identifier and each of said unique identifiers only identifying a single digital object in said fixed-content storage cluster;
identifying a first portion of said unique identifier, said first portion having a value within a range of all possible values;
accessing a table using said value indicated by said first portion of said unique identifier to identify a second computer node, said table mapping each of said all possible values to one of said computer nodes;
accessing an index on said second computer node using at least a second portion of said unique identifier to identify a third computer node, said index mapping each of said unique identifiers that include said value to at least one of said computer nodes; and
reading said digital object identified by said unique identifier from persistent storage on said third computer node and returning said digital object to said software application located external to said storage cluster.
2. The method as recited in claim 1 wherein said storage cluster is a write-once read-many (WORM) fixed-content storage cluster.
3. The method as recited in claim 1 wherein said index maps said at least a second portion of said unique identifier to a plurality of candidate computer nodes, said method further comprising:
determining whether a disk index of one of said candidate computer nodes includes said unique identifier; and
determining that said one of said candidate computer nodes is said third computer node.
4. The method as recited in claim 1 wherein said index maps the unique identifiers within said first range of addresses by listing partial unique identifiers.
5. The method as recited in claim 1 wherein said index maps each unique identifier directly to a volume identifier identifying a disk within said storage cluster, said method further comprising:
mapping said volume identifier to said one of said computer nodes, whereby said index maps each unique identifier indirectly to said one of said computer nodes.
6. The method as recited in claim 1 wherein said table and said index are located in RAM.
7. The method as recited in claim 1 wherein said unique identifier is a universally unique identifier (UUID).