Distributed catalog service for multi-cluster data processing platform
A first portion of a distributed catalog service is implemented for a given one of a plurality of distributed processing node clusters associated with respective data zones, each of the clusters being configured to perform processing operations utilizing local data resources locally accessible within its corresponding data zone. The first portion of the distributed catalog service receives a request to identify for each of a plurality of data resources to be utilized by an application initiated in the given cluster whether the data resource is a local or remote data resource relative to the given cluster, and provides a response to the request. The first portion of the distributed catalog service in combination with additional portions implemented for respective additional ones of the distributed processing node clusters collectively provide the distributed catalog service with capability to resolve local or remote status of data resources in each of the data zones.
1. A method comprising:
implementing a first portion of a distributed catalog service for a given one of a plurality of distributed processing node clusters associated with respective data zones, each of the clusters being configured to perform processing operations utilizing local data resources locally accessible within its corresponding data zone;
receiving in the first portion of the distributed catalog service a request to identify for each of a plurality of data resources to be utilized by an application initiated in the given cluster whether the data resource is a local data resource or a remote data resource relative to the given cluster; and
providing from the first portion of the distributed catalog service a response to the request;
wherein the first portion of the distributed catalog service in combination with additional portions implemented for respective additional ones of the plurality of distributed processing node clusters collectively provide the distributed catalog service with capability to resolve local or remote status of data resources in the data zones of each of the clusters responsive to requests from any other one of the clusters;
wherein a given one of the portions of the distributed catalog service in conjunction with its initiation as a Yet Another Resource Negotiator (YARN) application is registered as a service with a service registry of a resource manager of the corresponding cluster; and
wherein the method is implemented by at least one processing device comprising a processor coupled to a memory.
2. The method of claim 1 wherein the plurality of distributed processing node clusters comprises respective YARN clusters.
3. The method of claim 1 wherein the clusters comprise respective geographically-distributed regional data centers each configured to perform analytics processing utilizing the locally accessible data resources of its corresponding data zone.
4. The method of claim 1 wherein implementing a first portion of the distributed catalog service comprises:
providing a browsing interface accessible to one or more clients; and
providing a resolving interface accessible to an application master component of the application;
wherein the application master component is configured to access the resolving interface of the distributed catalog service in order to determine for each of the plurality of data resources to be utilized by the application whether the data resource is a local data resource or a remote data resource relative to the given cluster.
5. The method of claim 1 wherein the distributed catalog service is distributed over the clusters with each of the clusters having visibility of only its corresponding one of the portions of the distributed catalog service.
6. The method of claim 1 wherein the portions of the distributed catalog service are implemented as respective YARN applications running on respective ones of the clusters.
7. The method of claim 1 wherein at least one of the portions of the distributed catalog service comprises a plurality of entries with each such entry comprising a meta-resource comprising information characterizing location and accessibility of a corresponding one of the data resources.
8. The method of claim 7 wherein the meta-resource for a given local data resource comprises a file path to a storage location of that local data resource in the corresponding cluster.
9. The method of claim 7 wherein the meta-resource for a given remote data resource comprises information identifying another cluster for which that data resource is a local data resource.
10. The method of claim 7 wherein the meta-resource further comprises information regarding transformation of the data resource into one or more designated formats.
11. The method of claim 7 wherein a resolving interface of the portion of the distributed catalog service returns a given meta-resource responsive to a request that includes a corresponding meta-resource identifier.
12. The method of claim 1 wherein the service registry of the resource manager of the cluster is utilized to identify at least one of a browsing interface and a resolving interface of the given portion of the distributed catalog service implemented within that cluster.
13. The method of claim 1 wherein at least one of the portions of the distributed catalog service is configured in accordance with a configuration object that is stored in a predetermined storage location of the corresponding cluster.
14. The method of claim 1 wherein the method is performed at least in part in a worldwide data node coupled to one or more of the clusters.
15. The method of claim 1 wherein the method is performed at least in part in a worldwide data node that comprises a processing node of a given one of the clusters.
16. A computer program product comprising a non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code when executed by at least one processing device causes said at least one processing device:
to implement a first portion of a distributed catalog service for a given one of a plurality of distributed processing node clusters associated with respective data zones, each of the clusters being configured to perform processing operations utilizing local data resources locally accessible within its corresponding data zone;
to receive in the first portion of the distributed catalog service a request to identify for each of a plurality of data resources to be utilized by an application initiated in the given cluster whether the data resource is a local data resource or a remote data resource relative to the given cluster; and
to provide from the first portion of the distributed catalog service a response to the request;
wherein the first portion of the distributed catalog service in combination with additional portions implemented for respective additional ones of the plurality of distributed processing node clusters collectively provide the distributed catalog service with capability to resolve local or remote status of data resources in the data zones of each of the clusters responsive to requests from any other one of the clusters; and
wherein a given one of the portions of the distributed catalog service in conjunction with its initiation as a Yet Another Resource Negotiator (YARN) application is registered as a service with a service registry of a resource manager of the corresponding cluster.
17. The computer program product of claim 16 wherein at least one of the portions of the distributed catalog service comprises a plurality of entries with each such entry comprising a meta-resource comprising information characterizing location and accessibility of a corresponding one of the data resources.
18. An apparatus comprising:
at least one processing device having a processor coupled to a memory;
wherein said at least one processor is configured:
to implement a first portion of a distributed catalog service for a given one of a plurality of distributed processing node clusters associated with respective data zones, each of the clusters being configured to perform processing operations utilizing local data resources locally accessible within its corresponding data zone;
to receive in the first portion of the distributed catalog service a request to identify for each of a plurality of data resources to be utilized by an application initiated in the given cluster whether the data resource is a local data resource or a remote data resource relative to the given cluster; and
to provide from the first portion of the distributed catalog service a response to the request;
wherein the first portion of the distributed catalog service in combination with additional portions implemented for respective additional ones of the plurality of distributed processing node clusters collectively provide the distributed catalog service with capability to resolve local or remote status of data resources in the data zones of each of the clusters responsive to requests from any other one of the clusters; and
wherein a given one of the portions of the distributed catalog service in conjunction with its initiation as a Yet Another Resource Negotiator (YARN) application is registered as a service with a service registry of a resource manager of the corresponding cluster.
19. The apparatus of claim 18 wherein the service registry of the resource manager of the cluster is utilized to identify at least one of a browsing interface and a resolving interface of the given portion of the distributed catalog service implemented within that cluster.
20. An information processing system comprising the plurality of distributed processing node clusters of claim 18 .