Object storage system with local transaction logs, a distributed namespace, and optimized support for user directories
The present invention relates to object storage systems that support hierarchical directories within a namespace manifest stored as an object. The namespace manifest is stored as namespace manifest shards that are determined based on a partial key derived from the full name of the referenced version manifest in each directory entry. Each storage server maintains a local transaction log that keeps track of changes to the namespace manifest in response to put transactions. The namespace manifest can be updated after each put transaction is acknowledged, which minimizes the number of write operations required for a put transaction. Updates to the namespace manifest are batched, distributed, and processed concurrently and in parallel by the storage servers that store the corresponding shards. This reduces the latency that would otherwise exist in the object storage cluster as a result of a put transaction. Updates to namespace manifest shards optionally occur in a batch process using MapReduce techniques.
1. A method for a storage server within an object storage system to process a put transaction for a new version of an object associated with a new version manifest chunk, wherein the storage server contains an old version of the object associated with an old version manifest chunk, the method comprising:
validating the new version manifest chunk;
saving the new version manifest chunk to persistent storage;
comparing a unique version identifier for the new version manifest with a unique version identifier for the old version manifest stored in a local name index and updating a local name index for the object's name if the unique version identifier for the new version manifest is more current than the unique version identifier for the old version manifest stored in the local name index;
creating an entry in the storage server's local transaction log recording the existence of the new version manifest;
sending an acknowledgement message for the put transaction; and
asynchronously processing the entry to update one or more namespace manifests of the object storage system;
wherein each transaction log entry specifying creation or deletion of version manifests is mapped into records to be distributed to namespace manifests, wherein the mapped records comprise inclusive mapped records and iterative mapped records; and
wherein the inclusive mapped records each comprises type, enclosing directory name, the remaining portion of the fully qualified object name which is not part of the enclosing directory name, unique version identifier, and cryptographic hash identifying token of the version manifest, wherein the type comprises data indicating a version manifest exists or data indicating a version manifest does not exist.
2. A method for a storage server within an object storage system to process a put transaction for a new version of an object associated with a new version manifest chunk, wherein the storage server contains an old version of the object associated with an old version manifest chunk, the method comprising:
validating the new version manifest chunk;
saving the new version manifest chunk to persistent storage;
comparing a unique version identifier for the new version manifest with a unique version identifier for the old version manifest stored in a local name index and updating a local name index for the object's name if the unique version identifier for the new version manifest is more current than the unique version identifier for the old version manifest stored in the local name index;
creating an entry in the storage server's local transaction log recording the existence of the new version manifest;
sending an acknowledgement message for the put transaction; and
asynchronously processing the entry to update one or more namespace manifests of the object storage system;
wherein each transaction log entry specifying creation or deletion of version manifests is mapped into records to be distributed to namespace manifests, wherein the mapped records comprise inclusive mapped records and iterative mapped records; and
wherein each iterative mapped record comprises type, enclosing directory name, and the immediate sub-directory name relative to the enclosing directory.
3. A method for a storage server within an object storage system to process a put transaction for a new version of an object associated with a new version manifest chunk, wherein the storage server contains an old version of the object associated with an old version manifest chunk, the method comprising:
validating the new version manifest chunk;
saving the new version manifest chunk to persistent storage;
comparing a unique version identifier for the new version manifest with a unique version identifier for the old version manifest stored in a local name index and updating a local name index for the object's name if the unique version identifier for the new version manifest is more current than the unique version identifier for the old version manifest stored in the local name index;
creating an entry in the storage server's local transaction log recording the existence of the new version manifest;
sending an acknowledgement message for the put transaction; and
asynchronously processing the entry to update one or more namespace manifests of the object storage system;
wherein each transaction log entry specifying creation or deletion of version manifests is mapped into records to be distributed to namespace manifests, wherein the mapped records comprise inclusive mapped records and iterative mapped records; and
wherein each mapped record is assigned to a specific namespace manifest by a configuration rule assigning subsets of the entire namespace to specific namespace manifests, and to a specific shard of that namespace manifest, by the cryptographic hash of the enclosing path name.
4. The method of claim 3 , wherein each of the mapped records assigned to a batch based upon its assigned namespace shard; and wherein the collected batches are eventually dispatched to and merged with a target namespace shard.
5. The method of claim 4 , wherein the storage server that stores a namespace shard performs the merge of the batches of mapped records to create a new chunk for the namespace shard.
6. The method of claim 4 , wherein each transaction log entry specifying creation or deletion of version manifests is mapped into back-reference edit records to be distributed to storage servers, wherein the back-reference edit records comprise:
the content hash identifying token (CHIT) of the referenced chunk;
the CHIT of the referencing manifest; and
the effective replication count required of this chunk by the referencing manifest, wherein an effective replication count of zero indicates that the chunk is no longer required by this referencing manifest.
7. The method of claim 6 , wherein each storage server assigns the generated back-reference edit records to a corresponding target Negotiating Group and dispatches the resulting batch of back-reference edit records to the target Negotiating Group.