Data analytics platform for stateful, temporally-augmented observability, explainability and augmentation in web-based interactions and other user media
A data modeling and analytics platform augments and annotates content captured from a user's online interactions and other documents. The data modeling and analytics platform is performed within a machine learning and artificial intelligence-based processing environment that enables observability, explainability, and data analytics for dynamic information discovery over time within a user library that includes files representing the online interactions and documents containing information of user interest.
1 . A method, comprising:
ingesting input data comprised of one or more files representing at least one business process workflow;
modeling the input data to analyze a content of the one or more files in one or more machine learning models and execute the one or more business process workflows, the one or more machine learning models configured to:
extract the content in the one or more files, wherein one or more embeddings are generated from the one or more files,
create at least one embeddings vector and applying a vector cosine similarity to the at least one embeddings vector, and
apply one or more knowledge graphs to the content, the one or more knowledge graphs including information from a plurality of external data sources that is integrated with the content one or more files to develop an explainability layer that annotates the content by constructing a structured representation of the content using data points extracted from the one or more files, and associating the data points with particular, domain-specific information inferred from the external data sources that is relative to the content;
developing one or more domain-specific neural networks from the content, the domain-specific neural networks configured to analyze the structured representation of the content and the domain-specific information inferred from the external data sources;
augmenting the one or more domain-specific neural networks from the explainability layer over time to create temporal annotations of user-identified information from the one or more files to execute the at least one business process workflow; and
autonomously extracting the user-identified information from the content in response to one or more prompts of a large language model, wherein the one or more prompts are configured from the content and based on the at least one business process workflow defined within an artificial intelligence-based agent and curated to achieve user-defined outcomes in the at least one business process workflow, and
executing the at least one business process workflow, the artificial intelligence-based agent automatically assembled to execute the at least one business process workflow based on the user-identified information, wherein the explainability layer enables a traceability in the one or more workflows between the one or more files and the user-defined outcomes, wherein the artificial intelligence-based agent maintains a stateful execution of the at least one business workflow outside of conversational interactions.
2 . The method of claim 1 , further comprising delivering one or more generative outputs representing the at least one business process workflow, wherein the generative outputs include one or more of short-segment video clips, short-segment audio clips, document, presentations, and reports.
3 . The method of claim 1 , wherein the one or more files include meeting records representing online user interactions are captured by an the artificial intelligence-based agent, wherein the artificial intelligence-based agent performs an automated capture of the online user interactions, and wherein the meeting records include audio files and one or both of images and video files.
4 . The method of claim 3 , wherein the artificial intelligence-based agent captures the one or more files from a hosted meeting platform.
5 . The method of claim 1 , wherein the one or more files further include electronic records of information of user interest for the at least one business process workflow, the electronic records obtained either from a user upload, or an automated capture by instruction from the user, and wherein the electronic records include one or more of audio files, video files, documents, pictures, messages, and websites.
6 . The method of claim 1 , further comprising identifying one or more emotion analytics in the content, the one or more emotion analytics including mood changes, scene changes, changes in team relationships, and discussion topics in which emotion in one or more participants of the online user interactions is detected.
7 . The method of claim 1 , wherein the artificial intelligence-based agent engages with the large language model to automatically perform tasks based on the user-identified information associated with the content in response to the one or more prompts to execute the at least one business process workflow.
8 . The method of claim 7 , wherein the large language model is coupled to the one or more knowledge graphs, the one or more knowledge graphs integrating the particular, domain-specific information from the external data sources that explains outputs of the large language model in the explainability layer.
9 . The method of claim 1 , further comprising identifying a language in the content, and translating the content into a different language desired by the user.
10 . A method, comprising:
capturing one or more files representing at least one business process workflow within a data modeling and analytics software platform that includes one or more machine learning models, the one or more machine learning models configured to execute the at least one business process workflow by:
extracting a content in the one or more files, wherein one or more embeddings are generated from the one or more files,
creating at least one embeddings vector and applying a vector cosine similarity to the at least one embeddings vector, and
developing an explainability layer that annotates the content by applying one or more knowledge graphs to the content, the one or more knowledge graphs including information from a plurality of external data sources that is integrated with the content, wherein a structured representation of the content is constructed using data points extracted from the one or more files, and the data points are associated with particular, domain-specific information inferred from the external data sources that is relative to the content;
developing one or more domain-specific neural networks from the content, the domain-specific neural networks configured to analyze the structured representation of the content and the domain-specific information inferred from the external data sources;
augmenting the one or more domain-specific neural networks from the explainability layer over time to create temporal annotations of user-identified information from the one or more files to execute the at least one business process workflow; and
autonomously extracting the user-identified information from the content in response to one or more prompts of a large language model, wherein the one or more prompts are configured from the content and based on the at least one business process workflow defined within an artificial intelligence-based agent, and curated to achieve user-defined outcomes in the at least one business process workflow, and
executing the at least one business process workflow, the artificial intelligence-based agent automatically assembled to execute the at least one business process workflow based on the user-identified information, wherein the explainability layer enables a traceability in the one or more workflows between the one or more files and the user-defined outcomes, wherein the artificial intelligence-based agent maintains a stateful execution of the at least one business workflow outside of conversational interactions.
11 . The method of claim 10 , further comprising delivering one or more generative outputs representing the at least one business process workflow, wherein the generative outputs include one or more of short-segment video clips, short-segment audio clips, document, presentations, and reports.
12 . The method of claim 10 , wherein the one or more files include meeting records representing online user interactions are captured by the artificial intelligence-based agent, wherein the artificial intelligence-based agent performs an automated capture of the online user interactions, and wherein the meeting records include audio files and one or both of images and video files.
13 . The method of claim 12 , wherein the artificial intelligence-based agent captures the online user interactions from a hosted meeting platform.
14 . The method of claim 10 , further comprising capturing electronic records of information of user interest for the at least one business process workflow that further comprise the one or more files, the electronic records captured either from a user upload, or an automated capture by instruction from the user, and wherein the electronic records include one or more of audio files, video files, documents, pictures, messages, and websites.
15 . The method of claim 10 , further comprising identifying one or more emotion analytics in the content, the one or more emotion analytics including mood changes, scene changes, changes in team relationships, and discussion topics in which emotion in one or more participants of the online user interactions is detected.
16 . The method of claim 10 , wherein the artificial intelligence-based agent engages with the large language model to automatically perform tasks based on the user-identified information associated with the content in response to the one or more prompts to execute the at least one business process workflow.
17 . The method of claim 16 , wherein the large language model is coupled to the one or more knowledge graphs, the one or more knowledge graphs integrating the particular, domain-specific information from the external data sources that explains outputs of the large language model in the explainability layer.
18 . The method of claim 10 , further comprising identifying a language in the content, and translating the content into a different language desired by a user.
19 . A system, comprising:
an automated capture environment within a data analytics and modeling platform that captures one or more files representing at least one business process workflow;
a plurality of machine learning models that are configured to augment a content in the one or more files to execute the at least one business process workflow;
a diarization service that extracts and analyzes the content in the one or more files, wherein one or more embeddings are generated from the one or more files;
a content identification engine that creates at least one embeddings vector and applies a vector cosine similarity to the at least one embeddings vector;
one or more knowledge graphs that are applied to the content, the one or more knowledge graphs including information from a plurality of external data sources that is integrated with the content in the one or more files to develop an explainability layer that annotates the content by constructing a structured representation of the content using data points extracted from the one or more files, and associating the data points with particular, domain-specific information inferred from the external data sources that is relative to the content;
one or more domain-specific neural networks developed from the content, the domain-specific neural networks configured to analyze the structured representation of the content and the domain-specific information inferred from the external data sources; and
an artificial intelligence-based agent that autonomously extracts user-identified information from the content in response to one or more prompts of a large language model, wherein the one or more prompts are configured from the content based on the at least one business process workflow defined within the artificial intelligence-based agent and curated to achieve user-defined outcomes in the at least one business process workflow,
the artificial intelligence-based agent automatically assembled to execute the at least one business process workflow based on the user-identified information, wherein the explainability layer enables a traceability in the one or more workflows between the one or more files and the user-defined outcomes, and
wherein the one or more domain-specific neural networks are augmented by the explainability layer over time to create temporal annotations of user-identified information from the one or more files to execute the at least one business workflow, wherein the artificial intelligence-based agent maintains a stateful execution of the at least one business workflow outside of conversational interactions.
20 . The system of claim 19 , wherein one or more generative outputs representing the at least one business process workflow are delivered to the user, the generative outputs including one or more of short-segment video clips, short-segment audio clips, document, presentations, and reports.
21 . The system of claim 19 , wherein the meeting records representing the online user interactions are captured by the artificial intelligence-based agent, wherein the artificial intelligence-based agent performs an automated capture of the online user interactions, and wherein the meeting records include audio files and one or both of images and video files.
22 . The system of claim 21 , wherein the artificial intelligence-based agent captures the one or more files from a hosted meeting platform.
23 . The system of claim 19 , wherein the data analytics and modeling platform further captures electronic records of information of user interest for the at least one business process workflow, the electronic records obtained either from a user upload, or an automated capture by instruction from a user, and wherein the electronic records include one or more of audio files, video files, documents, pictures, messages, and websites.
24 . The system of claim 19 , further comprising an emotion detection engine that identifies one or more emotion analytics in the content, the one or more emotion analytics including mood changes, scene changes, changes in team relationships, and discussion topics in which emotion in one or more participants of the online user interactions is detected.
25 . The system of claim 19 , wherein the artificial intelligence-based agent engages with the large language model to automatically perform tasks based on the user-identified information associated with the content in response to the one or more prompts to execute the at least one business process workflow.
26 . The system of claim 25 , wherein the large language model is coupled to the one or more knowledge graphs, the one or more knowledge graphs integrating the particular, domain-specific information from the external data sources that explains outputs of the large language model in the explainability layer.
27 . The system of claim 19 , wherein the diarization service identifies a language in the content, and translates the content into a different language desired by the user.