Elimination of duplicate objects in storage clusters
Digital objects within a fixed-content storage cluster use a page mapping table and a hash-to-UID table to store a representation of each object. For each object stored within the cluster, a record in the hash-to-UID table stores the object's hash value and its unique identifier (or portions thereof). To detect a duplicate of an object, a portion of its hash value is used as a key into the page mapping table. The page mapping table indicates a node holding a hash-to-UID table indicating currently stored objects in a particular page range. Finding the same hash value but with a different unique identifier in the table indicates that a duplicate of an object exists. Portions of the hash value and unique identifier may be used in the hash-to-UID table. Unneeded duplicate objects are deleted by copying their metadata to a manifest and then redirecting unique identifiers to point at the manifest.
1. A method of deleting a duplicate of a first digital object within a storage cluster, said method comprising:
receiving a first unique identifier that identifies the location of a first digital object within said storage cluster;
receiving a second unique identifier that identifies the location of a second digital object within said storage cluster, wherein said digital objects being duplicates;
storing metadata associated with said second digital object in association with metadata associated with said first digital object in a metadata storage location;
creating a reference associated with said metadata storage location that identifies said location of said first digital object;
deleting said second digital object from said storage cluster; and
redirecting said second unique identifier such that said second unique identifier now identifies said metadata storage location, whereby said second unique identifier identifies said first digital object via said metadata storage location and said reference.
2. The method as recited in claim 1 wherein said deleting is performed by overwriting said second digital object, reclaiming disk space occupied by said second digital object, or redirecting said second unique identifier.
3. The method as recited in claim 1 further comprising:
accessing a page mapping table using a value indicated by said portion of a hash value of said second unique identifier to identify a computer node of said storage cluster;
accessing a hash table of said computer node to find an entry corresponding to said hash value and said second unique identifier; and
deleting said entry, whereby said hash table no longer includes a representation that said second digital object is present within said storage cluster.