Aggregating data ingested from disparate sources for processing using machine learning models
Presented herein are systems and methods for aggregating data from disparate sources to output information. A computing system may transform a first plurality of datasets of a plurality of data sources by converting a first format of the corresponding data source for each of the first plurality of datasets to generate a second plurality of datasets in a second format of the computing system. The computing system may identify, from the second plurality of datasets, a subset of datasets using a feature selected for evaluation of a utility of the feature. The computing system may apply a machine learning model configured for the selected feature to the subset of datasets to generate an output that measures a likelihood of usefulness. The computing system may cause a visualization of the output for the feature to be displayed for presentation on a dashboard interface based on a template configured for the feature.
1. A method of identifying artificial intelligence (AI) models to detect vulnerabilities in network environments, comprising:
retrieving, by one or more processors, a first plurality of datasets generated by a plurality of data sources associated with the network environment over a first time period;
identifying, by the one or more processors and from the first plurality of datasets, at least one dataset corresponding to a process selected from a plurality of processes;
selecting, by the one or more processors and from a plurality of AI models, an AI model based on the process of the plurality of processes, wherein the AI model is trained using a second plurality of datasets for the process over a second time period;
applying, by the one or more processors, the AI model to the at least one dataset corresponding to the process;
generating, by the one or more processors and based on applying the AI model to the at least one dataset, an output including a risk level indicating a degree of vulnerability of the process to failure; and
causing, by the one or more processors, presentation of the output including the risk level indicating the degree of vulnerability of the process to failure via a user interface.
2. The method of claim 1 , further comprising:
retrieving, by the one or more processors, a third plurality of datasets from the plurality of data sources over a third time period;
selecting, by the one or more processors, from the plurality of AI models, a subset of AI models corresponding to a subset of processes from the plurality of processes present in the third plurality of datasets; and
training, by the one or more processors, the subset of AI models using at least a portion of the third plurality of datasets.
3. The method of claim 2 , further comprising
identifying, by the one or more processors, the portion of the third plurality of datasets to use for training the subset of AI models and a second portion of the third plurality of datasets to use for validating the subset of AI models; and
validating, by the one or more processors, at least one of the subset of AI models using the second portion of the third plurality of datasets.
4. The method of claim 1 , further comprising:
determining, by the one or more processors, that data correction is to be performed on the at least one dataset based on one or more values of the at least one dataset; and
identifying, by the one or more processors, supplemental data from the network environment to add to the at least one dataset, responsive to determining that data correction is to be performed.
5. The method of claim 1 , further comprising generating, by the one or more processors, for each dataset of the plurality of datasets, a tag indicating a category of a plurality of categories for the plurality of processes to be evaluated, the plurality of categories including at least one of the risk level, a usage, a health, or a performance;
wherein identifying the at least one dataset further comprises identifying, from the plurality of datasets, the at least one dataset associated with the tag indicating the risk level.
6. The method of claim 1 , further comprising
providing, by the one or more processors, the user interface comprising a plurality of user interface elements corresponding to the plurality of processes; and
selecting, by the one or more processors, the process from the plurality of processes, responsive to interaction with a user interface elements of the plurality of user interface elements corresponding to the process.
7. The method of claim 1 , further comprising identifying, by the one or more processors, from a plurality of templates, a template with which to display the output based on the process,
wherein causing the presentation further comprises causing displaying of the output including the risk level indicating the degree of vulnerability of the process via the user interface.
8. The method of claim 1 , further comprising converting, by the one or more processors, the least one dataset of the first plurality of datasets from a first format to a second format, the first format used by at least one of the plurality of data sources to generate the at least one dataset, the second format compatible as input for the AI model.
9. The method of claim 1 , further comprising maintaining, by the one or more processors, the plurality of AI models comprising at least one of (i) a first AI model configured to generate a first output indicating usefulness of the process, (ii) a second AI model configured to generate a second output indicating performance metric, (iii) a third AI model configured to generate a third output indicating a health level, or (iv) the AI model configured to generate the output, each AI model of the plurality of AI models corresponding to a respective process of the plurality of processes in network environment.
10. The method of claim 1 , further comprising addressing a risk in the network environment based on the output including the risk level indicating the degree of the vulnerability of the process to failure.
11. A system for identifying artificial intelligence (AI) models to detect vulnerabilities in network environments, comprising:
one or more processors coupled with memory, configured to:
retrieve a first plurality of datasets generated by a plurality of data sources associated with the network environment over a first time period;
identify, from the first plurality of datasets, at least one dataset corresponding to a process selected from a plurality of processes;
select, from a plurality of AI models, an AI model based on the process of the plurality of processes, wherein the AI model is trained using a second plurality of datasets for the process over a second time period;
apply the AI model to the at least one dataset corresponding to the process;
generate, based on applying the AI model to the at least one dataset, an output including a risk level indicating a degree of vulnerability of the process to failure; and
cause presentation of the output including the risk level indicating the degree of vulnerability of the process to failure via a user interface.
12. The system of claim 11 , wherein the one or more processors are further configured to:
retrieve a third plurality of datasets from the plurality of data sources over a third time period;
select, from the plurality of AI models, a subset of AI models corresponding to a subset of processes from the plurality of processes present in the third plurality of datasets; and
train the subset of AI models using at least a portion of the third plurality of datasets.
13. The system of claim 12 , wherein the one or more processors are further configured to:
identify the portion of the third plurality of datasets to use for training the subset of AI models and a second portion of the third plurality of datasets to use for validating the subset of AI models; and
validate at least one of the subset of AI models using the second portion of the third plurality of datasets.
14. The system of claim 11 , wherein the one or more processors are further configured to:
determine that data correction is to be performed on the at least one dataset based on one or more values of the at least one dataset; and
identify supplemental data from the network environment to add to the at least one dataset, responsive to determining that data correction is to be performed.
15. The system of claim 11 , wherein the one or more processors are further configured to:
generate, for each dataset of the plurality of datasets, a tag indicating a category of a plurality of categories for the plurality of processes to be evaluated, the plurality of categories including at least one of the risk level, a usage, a health, or a performance; and
identify, from the plurality of datasets, the at least one dataset associated with the tag indicating the risk level.
16. The system of claim 11 , wherein the one or more processors are further configured to:
provide the user interface comprising a plurality of user interface elements corresponding to the plurality of processes; and
select the process from the plurality of processes, responsive to interaction with a user interface elements of the plurality of user interface elements corresponding to the process.
17. The system of claim 11 , wherein the one or more processors are further configured to:
identify, from a plurality of templates, a template with which to display the output based on the process,
cause displaying of the output including the risk level indicating the degree of vulnerability of the process via the user interface.
18. The system of claim 11 , wherein the one or more processors are further configured to convert the least one dataset of the first plurality of datasets from a first format to a second format, the first format used by at least one of the plurality of data sources to generate the at least one dataset, the second format compatible as input for the AI model.
19. The system of claim 11 , wherein the one or more processors are further configured to maintain the plurality of AI models comprising at least one of (i) a first AI model configured to generate a first output indicating usefulness of the process, (ii) a second AI model configured to generate a second output indicating performance metric, (iii) a third AI model configured to generate a third output indicating a health level, or (iv) the AI model configured to generate the output, each AI model of the plurality of AI models corresponding to a respective process of the plurality of processes in network environment.
20. The system of claim 11 , wherein a risk in the network environment is addressed based on the output including the risk level indicating the degree of the vulnerability of the process to failure.