Gen AI-based improved end-to-end data analytics tool
View Patent ↗A generative artificial intelligence-based system and method for providing an improved end-to-end data analytics tool is provided. Data from input unit(s) associated with multiple data sources located at disparate locations is collected. A data quality assessment is performed based on one or more pre-determined criteria. Transformed version of the collected data is processed for analyzing one or more data parameters associated with the transformed data to determine relationships and patterns within the transformed data. Prompts are generated related to operational issues associated with the specific domain. The prompts are provided to Large Language Models (LLMs) as input for generating diagnostic data and insights related to the operational issues. An optimized value of one or more modifiable prompt parameters associated with the generated prompts is determined for customizing the LLMs. Domain specific recommendations are provided by LLM based on the generated diagnostic data and insights for resolving the operational issues.
1 . A generative artificial intelligence-based system for providing an improved end-to-end data analytics tool, the system comprising:
a memory storing program instructions;
a processor executing program instructions stored in the memory; and
a data analytics engine executed by the processor and configured to:
collect data from input units associated with multiple data sources located at disparate locations, wherein a data quality assessment is performed based on one or more pre-determined criteria;
transform the collected data using one or more data transformation operations based on the data quality assessment;
determine relationships and patterns within the transformed data by analyzing one or more data parameters associated with the transformed data;
generate prompts related to operational issues associated with a specific domain using one or more prompt generation techniques and one or more prompt generation templates based on the determined relationships and patterns within the transformed data, wherein the prompts are provided to a Large Language Models (LLM) as an input for generating outputs comprising diagnostic data and insights related to the operational issues;
generate one or more domain specific recommendations by the LLM based on the generated outputs for resolving the operational issues; and
determine an optimal value of one or more modifiable prompt parameters associated with the generated prompts based on an iterative testing of the prompts and the generated outputs for customizing the LLM to generate the outputs relevant to the specific domain.
2 . The system as claimed in claim 1 , wherein the collected data is present in multiple formats comprising structured data format and unstructured data format, and wherein a reference tag associated with each of the multiple data sources is fetched by a data collection unit along with collected data for providing integrity and traceability of the collected data.
3 . The system as claimed in claim 2 , wherein a data analytics engine comprises a data loading and ingestion unit executed by a processor and is configured to extract one or more data elements from the collected data relating to the structured data format and the unstructured data format for loading and ingesting the extracted data, the one or more data elements associated with the collected data are loaded and ingested in a pre-determined data format.
4 . The system as claimed in claim 3 , wherein the data analytics engine comprises a data transformation and storage unit executed by the processor and configured to implement the one or more data transformation operations on the ingested data for transforming the collected data, and wherein the one or more data transformation operations comprise data cleaning, data validating, and data enrichment, and wherein the one or more data transformation operations are implemented by determining data associated with the specific domain, mapping modifications in the data, extracting data, transforming data, reviewing the transformed data, and storing the transformed data.
5 . The system as claimed in claim 4 , wherein the data transformation and storage unit performs a data tracking operation during data storage based on pre-determined data tracking parameters comprising recording data source name and data collection date, and wherein the data tracking operation is performed for tracking data origin, and details of data collection process, and the implemented one or more data transformation operations, and wherein the data transformation and storage unit employs one or more LLMs by communicating with an LLM unit for processing one or more pre-defined prompt queries that are executed offline to provide tracking of the multiple data sources.
6 . The system as claimed in claim 5 , wherein the data transformation and storage unit carries out continuous monitoring of the multiple data sources, and wherein refreshing of data is initiated periodically by the data transformation and storage unit to collect current and up-to-date data from the multiple data sources, and wherein the LLM is retrained along with refreshing of the data and the pre-determined data tracking parameters.
7 . The system as claimed in claim 1 , wherein the data analytics engine comprises a data processing and analysis unit executed by the processor and configured to transform the collected data and determine the relationships and the patterns within the transformed data by implementing one or more statistical and logical techniques.
8 . The system as claimed in claim 1 , wherein the data analytics engine comprises a prompt generation unit executed by the processor and configured to receive one or more user inputs in the form of user queries from an input unit for generating the prompts based on the user queries, the user queries relate to determining one or more operational parameters, wherein the prompt generation unit converts the user queries into one or more tokens by implementing a tokenization operation, and wherein the prompt generation unit fetches the transformed data from a data processing and analysis unit and associates the transformed data with the generated tokens by employing a Retrieval Augmented Generation (RAG) technique.
9 . The system as claimed in claim 8 , wherein the prompts along with one or more prompt responses are provided to the LLMs associated with a LLM unit as an input by the prompt generation unit for training the LLM and generating one or more outputs comprising one or more diagnostics data, determining patterns related to causes of performance issues, generating insights related to operational efficiencies and potential remedial actions to improve performance.
10 . The system as claimed in claim 8 , wherein the prompt generation unit generates one or more vector embeddings to capture specific and relevant domain knowledge from the multiple data sources, the one or more vector embeddings are used to enhance the one or more user inputs provided as the user queries to generate an augmented and enriched prompt comprising data from the multiple data sources, and wherein each of the one or more vector embeddings is an array of numerical values comprising a token ID number and a token name, the vector embeddings captures user behavioral features comprising semantic meaning, role and sentiment.
11 . The system as claimed in claim 10 , wherein the prompt generation unit processes the one or more vector embeddings to:
generate the prompts by using the one or more prompt generation techniques and the one or more prompt generation templates, the prompt generation techniques comprises an unsupervised learning technique and a semi-supervised learning technique to develop;
generate and optimize prompts to enhance output of the LLM, and wherein the one or more prompts generation templates are associated with pre-determined prompt elements comprising instructions, context, output indicator, and expected output with respect to the one or more operational parameters, the prompt elements are selected optionally depending upon the nature and purpose of the prompt.
12 . The system as claimed in claim 11 , wherein the prompt generation unit develops, generates, and optimizes prompts to enhance output of the LLMs for determining trends, correlations, patterns, outliers, and variations associated with the transformed data, and wherein the correlations, patterns, outliers, and variations associated with the collected data are determined by using one or more data analysis techniques, comprising a descriptive analysis technique, a diagnostic analysis technique, a predictive analysis technique, and a prescriptive analysis technique.
13 . The system as claimed in claim 1 , wherein the data analytics engine comprises a LLM unit executed by the processor and configured to receive the generated prompts from a prompt generation unit, and wherein the LLMs associated with the LLM unit are trained using the prompts for generating the diagnostic data and insights with respect to the one or more operational issues, and wherein the LLM unit provides the one or more domain specific recommendations relating to the diagnostic data and insights based on the generated prompts.
14 . The system as claimed in claim 13 , wherein the LLM unit improves performance of the LLMs for enhancing response quality of the LLMs by employing the one or more modifiable prompt parameters that control the randomness and diversity of output provided by the LLMs, and wherein the LLM unit determines the optimal value of the one or more modifiable prompt parameters based on the iterative testing of various prompts and generated responses, and the optimal value of the one or more modifiable prompt parameters are used for customizing the LLMs for generating accurate, context rich and domain relevant LLM responses.
15 . The system as claimed in claim 1 , wherein the data analytics engine comprises a data visualization unit executed by the processor and configured to receive the generated prompts from a prompt generation unit for providing visualization of one or more prompt elements associated with the prompts via an output unit, and wherein the data visualization unit provides visualization of the one or more domain specific recommendations relating to diagnostic data and insights generated by a LLM unit by processing the prompts, and wherein the data visualization unit provides visualization of the one or more domain specific recommendations for resolving operational issues via the output unit.
16 . The system as claimed in claim 15 , wherein the data analytics engine comprises a data visualization unit configured to provide visualization of the one or more operational parameters in one or more visualization forms comprising charts, graphs, maps, pie charts, bar charts, histograms, line graphs and dashboards on a Graphical User Interface (UI) associated with the output unit by using one or more Business Intelligence (BI) tools and a visualization tool.
17 . The system as claimed in claim 16 , wherein the data visualization unit communicates with thea LLM unit for providing inference for each of the one or more operational parameters in the form of summary and observations by processing the one or more visualization forms associated with the one or more operational parameters.
18 . The system as claimed in claim 16 , wherein the data visualization unit in communication with the LLM unit provides answers to one or more user queries along with targeted remedial initiatives depending on a problem area associated with the operational parameters, and wherein the data visualization unit in communication with the LLM unit generates advisory reports along with a magnitude of remedial impact for generating remedial decisions.
19 . A generative artificial intelligence-based method for providing an improved end-to-end data analytics tool, the method is implemented by a processor executing instructions stored in a memory, the method comprises:
collecting data from input unit(s) associated with multiple data sources located at disparate locations, wherein a data quality assessment is performed based on one or more pre-determined criteria;
transforming the collected data using one or more data transformation operations based on the data quality assessment;
determining relationships and patterns within the transformed data by analyzing one or more data parameters associated with the transformed data;
generating prompts related to operational issues associated with a specific domain using one or more prompt generation technique and one or more prompt generation templates based on the determined relationships and patterns within the transformed data, wherein the prompts are provided to a Large Language Models (LLMs) as an input for generating outputs comprising diagnostic data and insights related to the operational issues,
generating one or more domain specific recommendations by the LLM based on the generated outputs for resolving the operational issues; and
determining an optimizal value of one or more modifiable prompt parameters associated with the generated prompts based on an iterative testing of the prompts and the generated outputs for customizing the LLM to generate the outputs relevant to the specific domain.
20 . The method as claimed in claim 19 , wherein the one or more data transformation operations are implemented on the ingested data for transforming the collected data, and wherein the one or more data transformation operations comprise data cleaning, data validating, and data enrichment, and wherein the one or more data transformation operations are implemented by determining the data associated with the specific domain, mapping modifications in the data, extracting data, transforming data, reviewing the transformed data, and storing the transformed data.
21 . The method as claimed in claim 20 , wherein a data tracking operation is performed during data storage based on pre-determined data tracking parameters comprising recording data source name and data collection date, and wherein the data tracking operation is performed for tracking data origin, details of data collection process, and the data transformation operations, and wherein one or more LLMs are employed for processing one or more pre-defined prompt queries that are executed offline to provide tracking of the multiple data sources.
22 . The method as claimed in claim 19 , wherein one or more user inputs in the form of user queries are used for generating the prompts related to one or more user queries, wherein the user queries relate to determining one or more operational parameters, the user queries are converted into one or more tokens by implementing a tokenization operation, and wherein the transformed data is associated with the tokens by employing a Retrieval Augmented Generation (RAG) technique.
23 . The method as claimed in claim 22 , wherein the prompts along with one or more prompt responses are provided to the LLMs as an input for training the LLM and generating one or more outputs comprising one or more diagnostics data, determining patterns related to causes of performance issues, generating insights related to operational efficiencies and potential remedial actions to improve performance.
24 . The method as claimed in claim 23 , wherein one or more vector embeddings are generated to capture specific and relevant domain knowledge from the multiple data sources, the one or more vector embeddings are used to enhance the one or more user inputs provided as the user queries to generate an augmented and enriched prompt comprising data from the multiple data sources, and wherein each of the one or more vector embeddings is an array of numerical values comprising a token ID number and a token name, the vector embeddings captures user behavioral features comprising semantic meaning, role and sentiment.
25 . The method as claimed in claim 24 , wherein the one or more vector embeddings are processed for:
generating the prompts by using the one or more prompt generation techniques and the one or more prompt generation templates, the prompt generation techniques comprise an unsupervised learning technique and a semi-supervised learning technique to develop,
generate, and optimize prompts to enhance output of the LLM, and wherein the one or more prompts generation templates are associated with pre-determined prompt elements comprising instructions, context, output indicator, and expected output with respect to the one or more operational parameters, the prompt elements are selected optionally depending upon the nature and purpose of the prompt.
26 . The method as claimed in claim 25 , wherein performance of the LLM is improved for enhancing response quality of the LLM by employing the one or more modifiable prompt parameters that control the randomness and diversity of output provided by the LLM, and wherein the optimal value of the one or more modifiable prompt parameters is determined based on the iterative testing of various prompts and generated responses, and the optimal values of the one or more modifiable prompt parameters are used for customizing the LLM for generating accurate, context rich and domain relevant LLM responses.
27 . The method as claimed in claim 19 , wherein one or more prompt elements associated with the prompts are visualized, and wherein the visualization of the one or more domain specific recommendations relating to diagnostic data and insights is generated by processing the prompts, and wherein the one or more domain specific recommendations for resolving operational issues is visualized.
28 . The method as claimed in claim 27 , wherein answers to one or more user queries are provided along with targeted remedial initiatives depending on a problem area associated with the operational parameters, and wherein advisory reports are generated along with a magnitude of remedial impact for generating remedial decisions.
29 . A computer program product comprising:
a non-transitory computer-readable medium having computer program code stored thereon, the computer-readable program code comprising instructions that, when executed by a processor, causes the processor to:
collect data from input unit(s) associated with multiple data sources located at disparate locations, wherein a data quality assessment is performed based on one or more pre-determined criteria;
transform the collected data using one or more data transformation operations based on the data quality assessment;
determine relationships and patterns within the transformed data by analyzing one or more data parameters associated with the transformed data;
generate prompts related to operational issues associated with a specific domain using one or more prompt generation technique and one or more prompt generation templates based on the determined relationships and patterns within the transformed data, wherein the prompts are provided to a Large Language Model (LLM) as an input for generating outputs comprising diagnostic data and insights related to the operational issues;
generate one or more domain specific recommendations by the LLM based on the generated outputs for resolving the operational issues; and
determine an optimal value of one or more modifiable prompt parameters associated with the generated prompts based on an iterative testing of the prompts and the generated outputs for customizing the LLM to generate the outputs relevant to the specific domain.