Integration of distributed data processing platform with one or more distinct supporting platforms
An apparatus in one embodiment comprises at least one processing device having a processor coupled to a memory. The one or more processing devices are operative to configure a plurality of distributed processing nodes to communicate over a network, to obtain metadata characterizing data locally accessible in respective data zones of respective ones of the distributed processing nodes, and to populate catalog instances of a distributed catalog service for respective ones of the data zones utilizing the obtained metadata. Distributed data analytics are performed in the distributed processing nodes utilizing the populated catalog instances of the distributed catalog service and the locally accessible data of the respective data zones. The metadata characterizing the locally accessible data is illustratively obtained in a metadata repository from at least one of a master data management platform and a governance, risk and compliance platform.
1. A method comprising:
configuring a plurality of distributed processing nodes, each comprising a processor coupled to a memory, to communicate over a network;
obtaining metadata characterizing data locally accessible in respective data zones of respective ones of the distributed processing nodes;
populating catalog instances of a distributed catalog service for respective ones of the data zones utilizing the obtained metadata; and
performing distributed data analytics for a given analytics job in the distributed processing nodes utilizing the populated catalog instances of the distributed catalog service and the locally accessible data of the respective data zones;
wherein the given analytics job utilizes one or more first data items locally accessible within a first one of the data zones of a first one of the distributed processing nodes and one or more second data items locally accessible within at least a second one of the data zones of at least a second one of the distributed processing nodes;
wherein the obtained metadata comprises one or more metadata tags identifying at least one of the one or more first data items and the one or more second data items, a first entrance key permitting access to a first internal network of the first data zone, and a second entrance key permitting access to a second internal network of the second data zone;
wherein populating the catalog instances of the distributed catalog service further comprises provisioning a first populated catalog instance associated with the first data zone of the first distributed processing node with the first entrance key and provisioning a second populated catalog instance associated with the second data zone of the second distributed processing node with the second entrance key, and
wherein performing distributed data analytics for the given analytics job comprises at least one of:
the first populated catalog instance associated with the first data zone of the first distributed processing node utilizing at least one of the one or more metadata tags to map to one or more first physical storage locations of one or more of the first data items in one or more first data storage devices of the first distributed processing node and utilizing the first entrance key to access one or more of the first data items at the one or more first physical storage locations in the one or more first data storage devices of the first distributed processing node; and
the second populated catalog instance associated with the second data zone of the second distributed processing node utilizing at least one of the one or more metadata tags to map to one or more second physical storage locations of one or more of the second data items in one or more second data storage devices of the second distributed processing node and utilizing the second entrance key to access one or more of the second data items at the one or more second physical storage locations in the one or more second data storage devices of the second distributed processing node.
2. The method of claim 1 wherein obtaining metadata comprises obtaining at least a portion of the metadata from a master data management platform.
3. The method of claim 2 wherein metadata obtained from the master data management platform comprises at least one of location, identification and access control for each of a plurality of data items of the locally accessible data.
4. The method of claim 2 wherein the master data management platform is implemented at least in part on a processing platform that implements at least one of the distributed processing nodes.
5. The method of claim 2 wherein the distributed processing nodes comprise respective WWH nodes of a WWH platform that is at least partially integrated with the master data management platform.
6. The method of claim 2 further comprising providing at least one result of the distributed data analytics back to the master data management platform.
7. The method of claim 1 wherein obtaining metadata comprises obtaining at least a portion of the metadata from a governance, risk and compliance platform.
8. The method of claim 7 wherein metadata obtained from the governance, risk and compliance platform comprises at least one of a policy, a rule and a restriction for each of a plurality of data items of the locally accessible data.
9. The method of claim 7 further comprising one or more of:
providing at least one result of the distributed data analytics back to the governance, risk and compliance platform; and
providing at least one of audit information, log information and notification information from at least one of the distributed processing nodes back to the governance, risk and compliance platform.
10. The method of claim 7 wherein the metadata comprises at least a subset of:
an external address of one of the distributed processing nodes corresponding to a given one of the data zones;
governance, risk and compliance information for use in conjunction with a catalog instance associated with the given data zone; and
an internal address of a data processing location within the internal network of the given data zone;
wherein one or more analytics results generated using the catalog instance in the data processing location of the given data zone are returned to the governance, risk and compliance platform from the given data zone.
11. The method of claim 1 wherein obtaining metadata comprises obtaining metadata in a metadata repository from at least one of a master data management platform and a governance, risk and compliance platform.
12. The method of claim 11 wherein the metadata repository is implemented at least in part within the master data management platform.
13. The method of claim 1 wherein a first catalog of the distributed catalog service configured to track a first set of resources within a first one of the data zones can directly access the first set of resources within the first data zone and wherein the first catalog is further configured to indirectly access via a second catalog of the distributed catalog service a second set of resources tracked by the second catalog within a second one of the data zones.
14. The method of claim 1 wherein at least a subset of the distributed processing nodes are implemented in respective distributed data processing clusters corresponding to respective ones of the data zones and wherein the distributed catalog service is implemented at least in part as a plurality of interacting catalog applications running on respective ones of the distributed processing node clusters.
15. A computer program product comprising a non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code when executed by at least one processing device causes said at least one processing device:
to configure a plurality of distributed processing nodes to communicate over a network;
to obtain metadata characterizing data locally accessible in respective data zones of respective ones of the distributed processing nodes;
to populate catalog instances of a distributed catalog service for respective ones of the data zones utilizing the obtained metadata; and
to perform distributed data analytics for a given analytics job in the distributed processing nodes utilizing the populated catalog instances of the distributed catalog service and the locally accessible data of the respective data zones;
wherein the given analytics job utilizes one or more first data items locally accessible within a first one of the data zones of a first one of the distributed processing nodes and one or more second data items locally accessible within at least a second one of the data zones of at least a second one of the distributed processing nodes;
wherein the obtained metadata comprises one or more metadata tags identifying at least one of the one or more first data items and the one or more second data items, a first entrance key permitting access to a first internal network of the first data zone, and a second entrance key permitting access to a second internal network of the second data zone;
wherein populating the catalog instances of the distributed catalog service further comprises provisioning a first populated catalog instance associated with the first data zone of the first distributed processing node with the first entrance key and provisioning a second populated catalog instance associated with the second data zone of the second distributed processing node with the second entrance key; and
wherein performing distributed data analytics for the given analytics job comprises at least one of:
the first populated catalog instance associated with the first data zone of the first distributed processing node utilizing at least one of the one or more metadata tags to map to one or more first physical storage locations of one or more of the first data items in one or more first data storage devices of the first distributed processing node and utilizing the first entrance key to access one or more of the first data items at the one or more first physical storage locations in the one or more first data storage devices of the first distributed processing node; and
the second populated catalog instance associated with the second data zone of the second distributed processing node utilizing at least one of the one or more metadata tags to map to one or more second physical storage locations of one or more of the second data items in one or more second data storage devices of the second distributed processing node and utilizing the second entrance key to access one or more of the second data items at the one or more second physical storage locations in the one or more second data storage devices of the second distributed processing node.
16. The computer program product of claim 15 wherein obtaining metadata comprises obtaining at least a portion of the metadata from a master data management platform.
17. The computer program product of claim 15 wherein obtaining metadata comprises obtaining at least a portion of the metadata from a governance, risk and compliance platform.
18. An apparatus comprising:
at least one processing device having a processor coupled to a memory;
wherein said at least one processor is operative:
to configure a plurality of distributed processing nodes to communicate over a network;
to obtain metadata characterizing data locally accessible in respective data zones of respective ones of the distributed processing nodes;
to populate catalog instances of a distributed catalog service for respective ones of the data zones utilizing the obtained metadata; and
to perform distributed data analytics for a given analytics job in the distributed processing nodes utilizing the populated catalog instances of the distributed catalog service and the locally accessible data of the respective data zones;
wherein the given analytics job utilizes one or more first data items locally accessible within a first one of the data zones of a first one of the distributed processing nodes and one or more second data items locally accessible within at least a second one of the data zones of at least a second one of the distributed processing nodes;
wherein the obtained metadata comprises one or more metadata tags identifying at least one of the one or more first data items and the one or more second data items, a first entrance key permitting access to a first internal network of the first data zone, and a second entrance key permitting access to a second internal network of the second data zone;
wherein populating the catalog instances of the distributed catalog service further comprises provisioning a first populated catalog instance associated with the first data zone of the first distributed processing node with the first entrance key and provisioning a second populated catalog instance associated with the second data zone of the second distributed processing node with the second entrance key, and
wherein performing distributed data analytics for the given analytics job comprises at least one of:
the first populated catalog instance associated with the first data zone of the first distributed processing node utilizing at least one of the one or more metadata tags to map to one or more first physical storage locations of one or more of the first data items in one or more first data storage devices of the first distributed processing node and utilizing the first entrance key to access one or more of the first data items at the one or more first physical storage locations in the one or more first data storage devices of the first distributed processing node; and
the second populated catalog instance associated with the second data zone of the second distributed processing node utilizing at least one of the one or more metadata tags to map to one or more second physical storage locations of one or more of the second data items in one or more second data storage devices of the second distributed processing node and utilizing the second entrance key to access one or more of the second data items at the one or more second physical storage locations in the one or more second data storage devices of the second distributed processing node.
19. The apparatus of claim 18 wherein obtaining metadata comprises obtaining at least a portion of the metadata from a master data management platform.
20. The apparatus of claim 18 wherein obtaining metadata comprises obtaining at least a portion of the metadata from a governance, risk and compliance platform.