PARALLEL AND TRANSPARENT TECHNIQUE FOR RETRIEVING ORIGINAL CONTENT THAT IS RESTRUCTURED IN A DISTRIBUTED OBJECT STORAGE SYSTEM
The present disclosure relates to distributed object storage systems and provides a parallel and transparent technique for retrieving restructured objects using original chunk references. The content retrieval technique disclosed herein may be implemented with parallel operations by multiple storage servers in the system. The retrieval is transparent in that an original reference, referred to as a CHIT, may still be used to retrieve the original content, such that a client requesting the original content need not be aware that the original content has been restructured. Other embodiments, aspects and features are also disclosed.
1 . A method for getting content from a distributed object storage system, the method comprising:
receiving, by a storage server of the distributed object storage system, a get request for an original chunk with original-format content of a specified type category, wherein the original chunk is identified by an original chunk identifier token (CHIT);
performing a search to find all key-value tuples (KVTs) of the specified category stored at the storage server which have a key that includes the original CHIT, wherein a set of KVTs resulting from the search includes an alternate-index KVT and an alternate chunk KVT, wherein the alternate-index KVT points to the alternate chunk KVT, and wherein the alternate chunk KVT points to alternate-format content; and
generating and sending a get response that includes the set of KVTs.
2 . The method of claim 1 , wherein the original-format content comprises an original-format payload, and the specified category comprises a payload category.
3 . The method of claim 1 , wherein the original-format content comprises an original-format version manifest, and the specified category comprises a version manifest category.
4 . The method of claim 1 , wherein the original-format content comprises an original-format content manifest, and the specified category comprises a content manifest category.
5 . The method of claim 1 , further comprising:
evaluating the get response along with get responses returned from other storage servers in a negotiating group for the original chunk to determine a rendezvous transfer from one or more storage servers in the negotiating group to a requesting server.
6 . The method of claim 5 , further comprising:
performing the rendezvous transfer from the one or more storage servers in the negotiating group to the requesting server;
receiving, by the requesting server, alternate-format content as a result of the rendezvous transfer; and
processing the alternate-format content to regenerate the original-format content.
7 . The method of claim 6 , wherein the alternate-format content comprises a compressed original-format content, and wherein said processing comprises decompressing the alternate-format content to obtain the original-format content.
8 . The method of claim 6 , wherein the alternate-format content comprises a delta content relative to a base content, and wherein said processing comprises obtaining the base content and reconstructing the original-format content by adding the delta content to the base content.
9 . The method of claim 6 , wherein the alternate-format content comprises a plurality of erasure encoded slices, and wherein said processing comprises reconstructing the original-format content from the plurality of erasure encoded slices.
10 . The method of claim 5 , further comprising:
using a client-consensus method in determining the rendezvous transfer such that said evaluating is performed by the requesting server.
11 . The method of claim 5 , further comprising:
using a cluster-consensus method in determining the rendezvous transfer such that said evaluating is performed by the storage servers in the negotiating group.
12 . A storage server of a distributed object storage system, the storage server comprising:
at least one storage device managed by the storage server;
at least one processor;
a network connection to communicatively interconnect the storage server with other storage servers in the distributed object storage system; and
computer-readable instruction code stored in the at least one storage device, wherein the computer-readable instruction code is executable by the at least one processor to manage the at least one storage device and communicate with the other storage servers,
wherein a set of key-value tuples is stored in the at least one storage device, each key-value tuple in the set including a searchable key and an associated value, and
wherein the computer-readable instruction code is configured to find alternate-format content by searching the set of key-value tuples for original-format content, wherein the alternate-format content is derived from the original-format content, and the original-format content has been released.
13 . The storage server of claim 12 , wherein said search is performed in a single indexing operation.
14 . The storage server of claim 12 , wherein the original-format content comprises an original-format payload.
15 . The storage server of claim 12 , wherein the original-format content comprises an original-format manifest.
16 . The storage server of claim 12 , wherein the computer-readable instruction code is further configured to generate and send a get response that includes relevant key-value tuples found by said searching.
17 . A distributed object storage system that stores objects in chunks, the system comprising:
a plurality of storage servers communicatively interconnected by a network; and
a plurality of gateways providing access to the distributed storage system,
wherein the system searches a negotiating group for an original chunk containing the original-format content but finds an alternate chunk containing alternate-format content, wherein the alternate-format content is derived from the original-format content.
18 . The system of claim 17 , wherein each of the plurality of storage servers stores a searchable key-value-tuple (KVT) index that includes a first KVT entry pointing to the original-format content, a second KVT entry pointing to the alternate-format content, and a third KVT entry that indicates a relation between the original-format content and the alternate-format content.