Real-time monitoring and reporting systems and methods for information access platform
A new ecosystem monitoring solution provides novel features including a dashboard service, a dashboard notifier, and a near real time query processor. The dashboard service can continuously aggregate, through the dashboard notifier and optionally dashboard agents, metadata from disparate ecosystem components of a complex computing platform or ecosystem. The metadata include metrics of crawling, data ingestion, and content enrichment activities and health information of the disparate ecosystem components. The metrics are processed with respect to a time window utilizing an expression tree dynamically constructed by the query processor. The query processor can navigate the expression tree to form collection models. Each collection has aggregation functions for aggregating a set of metrics specified in a view model. Responsive to a view request, the view model can be dynamically updated utilizing the collection model and communicated to a user device for rendition and presentation of a view through a dashboard user interface.
1. A method, comprising:
continuously aggregating metadata from disparate ecosystem components of an ecosystem, the metadata comprising metrics of crawling, data ingestion, and content enrichment activities and health information of the disparate ecosystem components of the ecosystem, the aggregating performed by a dashboard monitoring system running on a server machine operating in an enterprise computing environment, the dashboard monitoring system configured for monitoring the ecosystem, the dashboard monitoring system having collection models and view models;
processing, by the dashboard monitoring system with respect to a time window, the metadata thus aggregated from the disparate ecosystem components of the ecosystem, the processing including the dashboard monitoring system directing the metadata into the collection models with respect to the time window, each collection model corresponding to a view model for aggregating metrics specified in a filter definition of the corresponding view model, each collection model representing a functional data structure and having a set of aggregation functions for aggregating a set of metrics in the metadata in accordance with the corresponding view model of the view models for rendering a view for the set of metrics;
responsive to a view request from a dashboard user interface on a user device requesting the view for the set of metrics, dynamically updating the view model by updating the filter definition, the dynamically updating comprising utilizing the set of aggregation functions to aggregate metric values associated with the set of metrics in the metadata aggregated from the disparate ecosystem components of the ecosystem, the dynamically updating performed by the dashboard monitoring system; and
communicating the view model from the dashboard monitoring system to the dashboard user interface for rendering the view for the set of metrics in the dashboard user interface on the user device.
2. The method according to claim 1 , wherein the collection models are formed in memory.
3. The method according to claim 1 , further comprising:
receiving the metrics of crawling, data ingestion, and content enrichment activities and health information pushed from dashboard notifiers running in the disparate ecosystem components of the ecosystem or a portion thereof.
4. The method according to claim 1 , further comprising:
polling the metrics of crawling, data ingestion, and content enrichment activities and health information through dashboard agents communicatively connected to the disparate ecosystem components of the ecosystem or a portion thereof.
5. The method according to claim 1 , wherein the disparate ecosystem components of the ecosystem comprise at least two of a data ingestion pipeline, a social media crawler, a web crawler, a data crawler configured for an enterprise system, or a predictive content analytics module.
6. The method according to claim 5 , wherein the metrics comprise a number of documents enriched through the data ingestion pipeline, a number of messages or documents crawled by the social media or the web crawler, and a number of jobs completed by the data crawler.
7. The method according to claim 1 , wherein the processing further comprises:
dynamically constructing an expression tree in memory, the expression tree having nodes representing predicates; and
navigating the nodes of the expression tree to form the set of aggregation functions for each collection model.
8. The method according to claim 1 , wherein the disparate ecosystem components of the ecosystem comprise an application having a dashboard notifier library.
9. The method according to claim 8 , wherein the disparate ecosystem components of the ecosystem further comprise a third-party application running on a node having a dashboard agent.
10. A system, comprising:
a processor;
a non-transitory computer-readable medium; and
stored instructions translatable by the processor to perform:
continuously aggregating metadata from disparate ecosystem components of an ecosystem, the metadata comprising metrics of crawling, data ingestion, and content enrichment activities and health information of the disparate ecosystem components of the ecosystem, the aggregating performed by a dashboard monitoring system running on a server machine operating in an enterprise computing environment, the dashboard monitoring system configured for monitoring the ecosystem, the dashboard monitoring system having collection models and view models;
processing, by the dashboard monitoring system with respect to a time window, the metadata thus aggregated from the disparate ecosystem components of the ecosystem, the processing including the dashboard monitoring system directing the metadata into the collection models with respect to the time window, each collection model corresponding to a view model for aggregating metrics specified in a filter definition of the corresponding view model, each collection model representing a functional data structure and having a set of aggregation functions for aggregating a set of metrics in the metadata in accordance with the corresponding view model of the view models for rendering a view for the set of metrics;
responsive to a view request from a dashboard user interface on a user device requesting the view for the set of metrics, dynamically updating the view model by updating the filter definition, the dynamically updating comprising utilizing the set of aggregation functions to aggregate metric values associated with the set of metrics in the metadata aggregated from the disparate ecosystem components of the ecosystem, the dynamically updating performed by the dashboard monitoring system; and
communicating the view model from the dashboard monitoring system to the dashboard user interface for rendering the view for the set of metrics in the dashboard user interface on the user device.
11. The system according to claim 10 , wherein the collection models are formed in memory.
12. The system of claim 10 , wherein the stored instructions are further translatable by the processor to perform:
receiving the metrics of crawling, data ingestion, and content enrichment activities and health information pushed from dashboard notifiers running in the disparate ecosystem components of the ecosystem or a portion thereof.
13. The system of claim 10 , wherein the stored instructions are further translatable by the processor to perform:
polling the metrics of crawling, data ingestion, and content enrichment activities and health information through dashboard agents communicatively connected to the disparate ecosystem components of the ecosystem or a portion thereof.
14. The system of claim 10 , wherein the disparate ecosystem components of the ecosystem comprise at least two of a data ingestion pipeline, a social media crawler, a web crawler, a data crawler configured for an enterprise system, or a predictive content analytics module.
15. The system of claim 14 , wherein the metrics comprise a number of documents enriched through the data ingestion pipeline, a number of messages or documents crawled by the social media or the web crawler, and a number of jobs completed by the data crawler.
16. A computer program product having a non-transitory computer-readable storage medium that stores computer instructions translatable by a processor to perform:
continuously aggregating metadata from disparate ecosystem components of an ecosystem, the metadata comprising metrics of crawling, data ingestion, and content enrichment activities and health information of the disparate ecosystem components of the ecosystem, the aggregating performed by a dashboard monitoring system running on a server machine operating in an enterprise computing environment, the dashboard monitoring system configured for monitoring the ecosystem, the dashboard monitoring system having collection models and view models;
processing, by the dashboard monitoring system with respect to a time window, the metadata thus aggregated from the disparate ecosystem components of the ecosystem, the processing including the dashboard monitoring system directing the metadata into the collection models with respect to the time window, each collection model corresponding to a view model for aggregating metrics specified in a filter definition of the corresponding view model, each collection model representing a functional data structure and having a set of aggregation functions for aggregating a set of metrics in the metadata in accordance with the corresponding view model of the view models for rendering a view for the set of metrics;
responsive to a view request from a dashboard user interface on a user device requesting the view for the set of metrics, dynamically updating the view model by updating the filter definition, the dynamically updating comprising utilizing the set of aggregation functions to aggregate metric values associated with the set of metrics in the metadata aggregated from the disparate ecosystem components of the ecosystem, the dynamically updating performed by the dashboard monitoring system; and
communicating the view model from the dashboard monitoring system to the dashboard user interface for rendering the view for the set of metrics in the dashboard user interface on the user device.
17. The computer program product of claim 16 , wherein the collection models are formed in memory.
18. The computer program product of claim 16 , wherein the instructions are further translatable by the processor to perform:
receiving the metrics of crawling, data ingestion, and content enrichment activities and health information pushed from dashboard notifiers running in the disparate ecosystem components of the ecosystem or a portion thereof.
19. The computer program product of claim 16 , wherein the instructions are further translatable by the processor to perform:
polling the metrics of crawling, data ingestion, and content enrichment activities and health information through dashboard agents communicatively connected to the disparate ecosystem components of the ecosystem or a portion thereof.
20. The computer program product of claim 16 , wherein the disparate ecosystem components of the ecosystem comprise at least two of a data ingestion pipeline, a social media crawler, a web crawler, a data crawler configured for an enterprise system, or a predictive content analytics module.