Ingestion manager for analytics platform

1. An apparatus comprising:

an ingestion manager;

a plurality of ingestion engines associated with the ingestion manager; and

an analytics platform configured to receive data from the ingestion engines under the control of the ingestion manager, the analytics platform comprising a plurality of analytics workspaces;

wherein the ingestion manager is configured to interact with one or more of the ingestion engines in conjunction with providing data to the plurality of analytics workspaces of the analytics platform;

wherein the ingestion manager is configured to select different subsets of the ingestion engines for providing data to different ones of the analytics workspaces, and to automatically modify its selection of one or more of the subsets of the ingestion engines selected for use with the different ones of the analytics workspaces based at least in part on monitored usage of data in those analytics workspaces;

wherein the ingestion manager selects a first subset of the plurality of ingestion engines for providing data to a first one of the analytics workspaces and selects a second subset of the plurality of ingestion engines for providing data to a second one of the analytics workspaces, the second subset being different than the first subset;

wherein the ingestion manager further implements a processing layer that augments at least one designated ingestion process of a selected one of the ingestion engines relating to ingestion of data into a corresponding one of the analytics workspaces from multiple related sources;

wherein the ingestion manager comprises an ingestion manager web portal providing web-based user access to:

(i) one or more ingest configuration application programming interfaces for accessing a first set of one or more components of the processing layer, the first set of one or more components comprising an ingest task definition component, a source element, a sink element and an extractor element; and

(ii) one or more task execution application programming interfaces for accessing a second set of one or more components of the processing layer, the second set of one or more components comprising an execution framework, a task scheduling element, a monitoring element and a status element; and

wherein the ingestion manager, ingestion engines and analytics platform are implemented by one or more processing devices each comprising a processor coupled to a memory.

2. The apparatus of claim 1 wherein two or more of the analytics workspaces of the analytics platform are configured to receive data from respective potentially disjoint subsets of the ingestion engines under the control of the ingestion manager.

3. The apparatus of claim 1 wherein the ingestion manager is configured to implement data-as-a-service functionality for the plurality of analytics workspaces of the analytics platform.

4. The apparatus of claim 1 wherein the ingestion manager is configured to select the one or more ingestion engines for providing data to the plurality of analytics workspaces of the analytics platform based at least in part on one or more of data type, data volume, data latency, intended usage, and service level agreement requirements.

5. The apparatus of claim 1 wherein the ingestion manager is configured to maintain historical information relating to data usage in the plurality of analytics workspaces of the analytics platform and to adjust the selection of one or more of the ingestion engines for providing data to the plurality of analytics workspaces based at least in part on the historical information.

6. The apparatus of claim 1 wherein the ingestion manager is configured to monitor usage of ingested data in the plurality of analytics workspaces of the analytics platform, to identify related data responsive to the monitoring, and to control ingestion of the related data by the plurality of analytics workspaces.

7. The apparatus of claim 1 wherein the ingestion manager is configured to permit two or more of the analytics workspaces to register to receive data from respective potentially disjoint subsets of the ingestion engines.

8. The apparatus of claim 1 wherein the ingestion manager is configured to control the ingestion of data into operational data containers that are accessible to each of at least a subset of the analytics workspaces of the analytics platform.

9. The apparatus of claim 1 wherein the ingestion manager is configured to ensure compliance with service level agreements for data ingestion on behalf of the plurality of analytics workspaces of the analytics platform.

10. The apparatus of claim 1 wherein the ingestion manager is configured to control enforcement of one or more specified policies relating to ingestion of data on behalf of the plurality of analytics workspaces of the analytics platform.

11. The apparatus of claim 1 wherein the ingestion manager is configured to identify relationships between ingested data sets and to create linkages between the data sets based at least in part on the identified relationships.

12. The apparatus of claim 1 wherein the ingestion manager is configured to audit placement of data received by the plurality of analytics workspaces of the analytics platform from the ingestion engines.

13. The apparatus of claim 1 wherein the ingestion manager is configured for one or more of the following:

to orchestrate one or more characteristics of a data ingestion process implemented by at least one of the ingestion engines;

to select one or more of the ingestion engines for providing data to the plurality of analytics workspaces based at least in part on data transformation or data quality improvement capabilities of the one or more ingestion engines; and

to provide one or more data ingestion functions not provided by one or more of the ingestion engines.

14. A method comprising:

associating a plurality of ingestion engines with an ingestion manager; and

controlling in the ingestion manager receipt of data by an analytics platform from the ingestion engines, the analytics platform comprising a plurality of analytics workspaces;

wherein controlling receipt of data by the analytics platform from the ingestion engines comprises interacting with one or more of the ingestion engines in conjunction with providing data to the plurality of analytics workspaces of the analytics platform;

wherein the ingestion manager is configured to select different subsets of the ingestion engines for providing data to different ones of the analytics workspaces, and to automatically modify its selection of one or more of the subsets of the ingestion engines selected for use with the different ones of the analytics workspaces based at least in part on monitored usage of data in those analytics workspaces;

wherein the ingestion manager selects a first subset of the plurality of ingestion engines for providing data to a first one of the analytics workspaces and selects a second subset of the plurality of ingestion engines for providing data to a second one of the analytics workspaces, the second subset being different than the first subset;

wherein the ingestion manager further implements a processing layer that augments at least one designated ingestion process of a selected one of the ingestion engines relating to ingestion of data into a corresponding one of the analytics workspaces from multiple related sources;

wherein the ingestion manager comprises an ingestion manager web portal providing web-based user access to:

(i) one or more ingest configuration application programming interfaces for accessing a first set of one or more components of the processing layer, the first set of one or more components comprising an ingest task definition component, a source element, a sink element and an extractor element; and

(ii) one or more task execution application programming interfaces for accessing a second set of one or more components of the processing layer, the second set of one or more components comprising an execution framework, a task scheduling element, a monitoring element and a status element; and

wherein the associating and controlling are performed by at least one processing device comprising a processor coupled to a memory.

15. The method of claim 14 wherein controlling receipt of data by the analytics platform from the ingestion engines further comprises:

maintaining historical information relating to data usage in the plurality of analytics workspaces of the analytics platform; and

adjusting selection of one or more of the ingestion engines for providing data to the plurality of analytics workspaces based at least in part on the historical information.

16. The method of claim 15 wherein controlling receipt of data by the analytics platform from the ingestion engines further comprises at least one of:

permitting two or more of the analytics workspaces to register to receive data from respective potentially disjoint subsets of the ingestion engines;

controlling the ingestion of data into operational data containers that are accessible to each of at least a subset of the analytics workspaces;

ensuring compliance with service level agreements for data ingestion on behalf of the plurality of analytics workspaces;

controlling enforcement of one or more specified policies relating to ingestion of data on behalf of the plurality of analytics workspaces of the analytics platform;

identifying relationships between ingested data sets and creating linkages between the data sets based at least in part on the identified relationships; and

auditing placement of data received by the plurality of analytics workspaces of the analytics platform from the ingestion engines.

17. The method of claim 14 wherein the ingestion manager is configured to select the one or more ingestion engines for providing data to the plurality of analytics workspaces of the analytics platform based at least in part on one or more of data type, data volume, data latency, intended usage, and service level agreement requirements.

18. A computer program product comprising a non-transitory processor-readable storage medium having one or more software programs embodied therein, wherein the one or more software programs when executed by at least one processing device causes said at least one processing device:

to associate a plurality of ingestion engines with an ingestion manager; and

to control in the ingestion manager receipt of data by an analytics platform from the ingestion engines, the analytics platform comprising a plurality of analytics workspaces;

wherein controlling receipt of data by the analytics platform from the ingestion engines comprises interacting with one or more of the ingestion engines in conjunction with providing data to the plurality of analytics workspaces of the analytics platform;

wherein the ingestion manager is configured to select different subsets of the ingestion engines for providing data to different ones of the analytics workspaces, and to automatically modify its selection of one or more of the subsets of the ingestion engines selected for use with the different ones of the analytics workspaces based at least in part on monitored usage of data in those analytics workspaces;

wherein the ingestion manager selects a first subset of the plurality of ingestion engines for providing data to a first one of the analytics workspaces and selects a second subset of the plurality of ingestion engines for providing data to a second one of the analytics workspaces, the second subset being different than the first subset;

wherein the ingestion manager further implements a processing layer that augments at least one designated ingestion process of a selected one of the ingestion engines relating to ingestion of data into a corresponding one of the analytics workspaces from multiple related sources; and

wherein the ingestion manager comprises an ingestion manager web portal providing web-based user access to:

(i) one or more ingest configuration application programming interfaces for accessing a first set of one or more components of the processing layer, the first set of one or more components comprising an ingest task definition component, a source element, a sink element and an extractor element; and

(ii) one or more task execution application programming interfaces for accessing a second set of one or more components of the processing layer, the second set of one or more components comprising an execution framework, a task scheduling element, a monitoring element and a status element.

19. The computer program product of claim 18 wherein controlling receipt of data by the analytics platform from the ingestion engines further comprises:

maintaining historical information relating to data usage in the plurality of analytics workspaces of the analytics platform; and

adjusting selection of one or more of the ingestion engines for providing data to the plurality of analytics workspaces based at least in part on the historical information.

20. The computer program product of claim 18 wherein controlling receipt of data by the analytics platform from the ingestion engines further comprises at least one of:

permitting two or more of the analytics workspaces to register to receive data from respective potentially disjoint subsets of the ingestion engines;

controlling the ingestion of data into operational data containers that are accessible to each of at least a subset of the analytics workspaces;

ensuring compliance with service level agreements for data ingestion on behalf of the plurality of analytics workspaces;

controlling enforcement of one or more specified policies relating to ingestion of data on behalf of the plurality of analytics workspaces of the analytics platform;

identifying relationships between ingested data sets and creating linkages between the data sets based at least in part on the identified relationships; and

auditing placement of data received by the plurality of analytics workspaces of the analytics platform from the ingestion engines.

21. The computer program product of claim 18 wherein the ingestion manager is configured to select the one or more ingestion engines for providing data to the plurality of analytics workspaces of the analytics platform based at least in part on one or more of data type, data volume, data latency, intended usage, and service level agreement requirements.