Managing data transfer between endpoints in a distributed computing environment
A file fetcher manager provides commonly-utilized management and resource allocation for multiple file fetchers that each implement a different type of mechanism or protocol for transferring data files between peer endpoints in meshes associated with a cloud-computing service. Each file fetcher is configured with both client-side and server-side components to retrieve and serve out data files. The file fetcher manager encapsulates the file fetchers to provide an abstract interface to callers while hiding the underlying details of the file fetchers. The file fetcher manager is arranged for managing simultaneous operations of the multiple file fetchers to route requests from the callers to the appropriate file fetchers, and for scheduling work items for the file fetchers so that data transfers efficiently utilize available resources while keeping the entire transfer process coherent and organized.
1. A method for providing a file fetcher arranged for serving and retrieving files to and from endpoints on a network supporting peer-to-peer file transfer, the method comprising the steps of:
associating work items in a work list with pending file requests from a caller at an endpoint, the work list being arranged to support prioritization of the work items;
obtaining a list of endpoints that contain at least a portion of files associated with the pending file requests;
maintaining a state for a file, the state identifying the file as being partially or completely downloaded;
retrieving at least a portion of a file in accordance with the work list, the list of endpoints, and the file state; and
serving at least a portion of a file responsively to a request from a caller at a remote endpoint,
wherein said retrieving step further comprises downloading the file using chunked file fetching and includes the further steps of:
receiving a list of available file chunks at an endpoint, and
receiving, from the caller, a number of file chunks and associated size of each chunk for creating the file state for a file, wherein space of a required size is allocated, at the file fetcher, prior to retrieving the file, based upon said number of file chunks and associated size of each chunk received from the caller, and
wherein the work list contains a plurality of work items for a file so that the file is retrieved in chunks in parallel from a plurality of endpoints;
in which a bitmap is appended at the end of the file and is maintained in an in-memory cache; and
wherein the bitmap appended to the file is removed after a last chunk is written to the file.
2. The method of claim 1 including a further step of authorizing the remote endpoint to a core object from which the file portion will be served out to the remote caller.
3. The method of claim 1 in which the list of endpoints is provided by an enclosure locator service.
4. The method of claim 1 including the further steps of locking a chunk to prevent duplicate requests from the caller, and requesting additional file chunks.
5. The method of claim 4 including a further step of implementing RDC-based chunked file fetching.
6. The method of claim 1 including a further step of implementing an interface between the file fetcher and a file fetcher manager that is commonly utilized by a plurality of file fetchers that operate at each respective endpoint, the file fetcher manager routing requests among ones of the plurality of file fetchers.
7. A method performed by a local file fetcher manager for managing a plurality of local file fetchers, the method comprising the steps of:
exposing file retrieval and server functionality provided by the file fetchers to callers comprising a local caller at a local endpoint and to a remote caller at a remote peer endpoint, the local endpoint and the remote peer endpoint being coupled to a cloud service supporting peer-to-peer file sharing;
querying one or more remote file fetcher managers to determine capabilities of remote file fetchers at respective remote peer endpoints, wherein the capabilities indicate whether each queried remote file fetcher is RDC-enabled;
maintaining a list indicating the RDC-enabled capabilities of each remote file fetcher in a cache; and
routing requests from the callers to a file fetcher selected among the plurality of local file fetchers according to the capabilities,
wherein a file is retrieved in chunks in parallel from a plurality of endpoints, and
wherein a bitmap is appended at the end of the file and is maintained in an in-memory cache; and
wherein the bitmap appended to the file is removed after a last chunk is written to the file.
8. The method of claim 7 including a further step of transferring work for retrieving a file among the local file fetchers.
9. The method of claim 7 including a further step of scheduling work items associated with the plurality of file fetchers with a global thread pool.
10. The method of claim 7 including the further steps of rank-ordering the local file fetchers by preference and routing file fetch requests by preference.
11. The method of claim 7 including a further step of passing a pointer to a retrieved file to the local caller.