Scalable edge computing using devices with limited resources
An apparatus in one embodiment includes at least one processing device comprising a processor coupled to a memory. The processing device is configured to discover data sources associated with respective edge devices configured to communicate over at least one network, to obtain metadata characterizing the data sources, and to control performance of distributed analytics across a plurality of distributed processing nodes of the network utilizing the obtained metadata and locally accessible data provided by the corresponding data sources. In some embodiments, the edge devices comprise respective IoT gateways each associated with a set of IoT data sources. The IoT gateways are illustratively arranged in respective data zones that also encompass their respective sets of IoT data sources, and the distributed analytics is performed without requiring that locally accessible data from the IoT data sources leave the corresponding data zones. At least portions of the metadata may be obtained from an edge device management system associated with at least a subset of the edge devices.
1. A method comprising:
discovering data sources associated with respective edge devices configured to communicate over at least one network, each edge device comprising a processor coupled to a memory;
obtaining metadata characterizing the data sources, the obtained metadata comprising a metadata model characterizing the data sources as respective meta-resources each associated with at least one device profile and a set of one or more addressable characteristics, the meta-resources comprising names of the data sources and credentials for accessing the data sources, the device profiles comprising information associated with at least one of device types of the data sources and locations of the data sources;
specifying at least one of a given device type and a given location of data sources to be used in distributed analytics performed across a plurality of distributed processing nodes of a network;
utilizing the device profiles in the metadata model to select a subset of the data sources having said at least one of the given device type and the given location as data sources on which the distributed analytics is to be performed; and
controlling performance of the distributed analytics across the plurality of distributed processing nodes utilizing the meta-resources and the sets of addressable characteristics of the selected subset of the data sources in the metadata model to access locally accessible data provided by the selected subset of the data sources;
wherein controlling performance of the distributed analytics comprises:
receiving results of intermediate computations performed on respective ones of a plurality of datasets in respective ones of the distributed processing nodes;
performing at least one global computation based at least in part on the results of the intermediate computations;
repeating at least a subset of the receiving and performing in each of a plurality of iterations; and
passing a result of the global computation in a first one of the iterations as an input to the intermediate computations in a second one of the iterations.
2. The method of claim 1 wherein the edge devices comprise respective IoT gateways each associated with a set of IoT data sources.
3. The method of claim 2 wherein the IoT gateways are arranged in respective data zones that also encompass their respective sets of IoT data sources and wherein the distributed analytics is performed without requiring that locally accessible data from the IoT data sources leave the corresponding data zones.
4. The method of claim 1 wherein at least a subset of the distributed processing nodes are arranged in respective ones of a plurality of data zones in association with respective ones of the edge devices.
5. The method of claim 4 further comprising:
populating catalog instances of a distributed catalog service for respective ones of the data zones utilizing at least portions of the obtained metadata; and
utilizing the catalog instances in controlling performance of the distributed analytics across the distributed processing nodes.
6. The method of claim 5 wherein at least a subset of the distributed processing nodes are implemented in respective distributed processing node clusters corresponding to respective ones of the data zones and wherein the distributed catalog service is implemented at least in part as a plurality of interacting catalog applications running on respective ones of the distributed processing node clusters.
7. The method of claim 1 wherein the device profiles further comprise metadata characterizing one or more types of data collected by the data sources.
8. The method of claim 1 wherein at least portions of the metadata are obtained in at least near real-time relative to performing distributed analytics utilizing those portions of the obtained metadata.
9. The method of claim 1 wherein obtaining metadata comprises obtaining at least portions of the metadata from an edge device management system associated with at least a subset of the edge devices.
10. The method of claim 9 wherein a distributed catalog service utilized in discovering data sources and obtaining metadata characterizing the data sources is provided at least in part as one or more microservices of the edge device management system.
11. The method of claim 10 wherein the distributed catalog service operates in accordance with a lazy protocol in which the data sources are discovered and the metadata characterizing the data sources is obtained on an as-needed basis in accordance with the particular type of distributed analytics to be performed.
12. The method of claim 1 wherein one or more of the edge devices are part of a same distributed processing node cluster as one or more of the distributed processing nodes that perform the distributed analytics.
13. A computer program product comprising a non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code when executed by at least one processing device causes said at least one processing device:
to discover data sources associated with respective edge devices configured to communicate over at least one network, each edge device comprising a processor coupled to a memory;
to obtain metadata characterizing the data sources, the obtained metadata comprising a metadata model characterizing the data sources as respective meta-resources each associated with at least one device profile and a set of one or more addressable characteristics, the meta-resources comprising names of the data sources and credentials for accessing the data sources, the device profiles comprising information associated with at least one of device types of the data sources and locations of the data sources;
to specify at least one of a given device type and a given location of data sources to be used in distributed analytics performed across a plurality of distributed processing nodes of a network;
to utilize the device profiles in the metadata model to select a subset of the data sources having said at least one of the given device type and the given location as data sources on which the distributed analytics is to be performed; and
to control performance of the distributed analytics across the plurality of distributed processing nodes utilizing the meta-resources and the sets of addressable characteristics of the selected subset of the data sources in the metadata model to access locally accessible data provided by the selected subset of the data sources;
wherein controlling performance of the distributed analytics comprises:
receiving results of intermediate computations performed on respective ones of a plurality of datasets in respective ones of the distributed processing nodes;
performing at least one global computation based at least in part on the results of the intermediate computations;
repeating at least a subset of the receiving and performing in each of a plurality of iterations; and
passing a result of the global computation in a first one of the iterations as an input to the intermediate computations in a second one of the iterations.
14. The computer program product of claim 13 wherein the edge devices comprise respective IoT gateways each associated with a set of IoT data sources, the IoT gateways being arranged in respective data zones that also encompass their respective sets of IoT data sources, and wherein the distributed analytics is performed without requiring that locally accessible data from the IoT data sources leave the corresponding data zones.
15. The computer program product of claim 13 wherein obtaining metadata comprises obtaining at least portions of the metadata from an edge device management system associated with at least a subset of the edge devices.
16. An apparatus comprising:
at least one processing device comprising a processor coupled to a memory;
wherein said at least one processing device is configured:
to discover data sources associated with respective edge devices configured to communicate over at least one network, each edge device comprising a processor coupled to a memory;
to obtain metadata characterizing the data sources, the obtained metadata comprising a metadata model characterizing the data sources as respective meta-resources each associated with at least one device profile and a set of one or more addressable characteristics, the meta-resources comprising names of the data sources and credentials for accessing the data sources, the device profiles comprising information associated with at least one of device types of the data sources and locations of the data sources;
to specify at least one of a given device type and a given location of data sources to be used in distributed analytics performed across a plurality of distributed processing nodes of a network;
to utilize the device profiles in the metadata model to select a subset of the data sources having said at least one of the given device type and the given location as data sources on which the distributed analytics is to be performed; and
to control performance of the distributed analytics across the plurality of distributed processing nodes utilizing the meta-resources and the sets of addressable characteristics of the selected subset of the data sources in the metadata model to access locally accessible data provided by the selected subset of the data sources;
wherein controlling performance of the distributed analytics comprises:
receiving results of intermediate computations performed on respective ones of a plurality of datasets in respective ones of the distributed processing nodes;
performing at least one global computation based at least in part on the results of the intermediate computations;
repeating at least a subset of the receiving and performing in each of a plurality of iterations; and
passing a result of the global computation in a first one of the iterations as an input to the intermediate computations in a second one of the iterations.
17. The apparatus of claim 16 wherein the edge devices comprise respective IoT gateways each associated with a set of IoT data sources, the IoT gateways being arranged in respective data zones that also encompass their respective sets of IoT data sources, and wherein the distributed analytics is performed without requiring that locally accessible data from the IoT data sources leave the corresponding data zones.
18. The apparatus of claim 16 wherein obtaining metadata comprises obtaining at least portions of the metadata from an edge device management system associated with at least a subset of the edge devices.
19. The apparatus of claim 16 wherein at least a subset of the distributed processing nodes are arranged in respective ones of a plurality of data zones in association with respective ones of the edge devices.
20. The apparatus of claim 19 wherein said at least one processing device is further configured:
to populate catalog instances of a distributed catalog service for respective ones of the data zones utilizing at least portions of the obtained metadata; and
to utilize the catalog instances in controlling performance of the distributed analytics across the distributed processing nodes.