Accuracy and providing explainability and transparency for query response using machine learning models
The implementations herein disclose advanced systems and methods for integrating, analyzing, and reasoning over heterogeneous data at scale. In some implementations, the system comprises a synergistic data processing infrastructure featuring: a graph database core for unified data representation; specialized loaders for concurrent ingestion and processing of structured, unstructured, and time series data; a natural language reasoning engine leveraging large language models; and a multi-modal user interface.
1. A non-transitory, computer-readable storage medium comprising instructions recorded thereon, wherein the instructions when executed by at least one data processor of a computer system, cause the computer system to:
receive, via a user interface displayed at a user device, a query, wherein the query comprises natural language;
parse and translate, by the computer system, the natural language query into a graph traversal or database query operation;
perform, by the computer system, the graph traversal or database query operation on a graph database to identify one or more graph representations,
wherein the graph database comprises a plurality of graph representations defined by a graph schema, wherein each graph representation of the plurality of graph representations comprises:
a plurality of nodes, each node representing an entity;
a plurality of edges, each edge connecting at least two nodes and defining a relationship between at least two entities represented by the at least two nodes; and
a plurality of properties, each property defining an attribute of one or more entities or one or more relationships;
and wherein each graph representation of the plurality of graph representations is derived from a plurality of extracted data points, each extracted data point comprising at least one of: structured data, unstructured data, or time series data, wherein each extracted data point comprising unstructured data is assigned, using one or more machine learning models, to one or more of a plurality of categories based at least in part on a relationship between each extracted data point and at least one other extracted data point, wherein each extracted categorized data point comprising unstructured data is assigned to a cluster of a plurality of clusters based at least in part on a relationship of each extracted categorized data point to a characteristic of a source of the unstructured data, and wherein the assignment of categories and clusters to each extracted data point comprising unstructured data alters a structured data schema used to extract the structured data;
retrieve, by the computer system, the one or more graph representations of the plurality of graph representations;
score, by the computer system, the one or more graph representations based on a relevance of the one or more graph representations to the natural language query to identify at least one relevant graph representation;
generate, by the computer system, a response to the natural language query, wherein generating the response comprises providing the at least one relevant graph representation to a large language model (LLM), and wherein the response comprises:
natural language;
a visual representation of the graph traversal;
an indicator of one or more nodes of the plurality of nodes and one or more edges of the plurality of edges within the visual representation, wherein the one or more nodes and one or more edges are determined to be relevant to the query; and
a structured citation comprising metadata indicating a source associated with the one or more nodes and one or more edges; and
display, by the user the user interface on the user device, the response to the natural language query.
2. A computer system comprising:
at least one hardware processor; and
at least one non-transitory memory storing instructions, which, when executed by the at least one hardware processor, cause the computer system to:
receive, via a user interface displayed at a user device, a query, wherein the query comprises natural language;
parse and translate, by the computer system, the natural language query into a graph traversal or database query operation;
perform, by the computer system, the graph traversal or database query operation on a graph database to identify one or more graph representations, wherein the graph database comprises a plurality of graph representations defined by a graph schema, wherein each graph representation of the plurality of graph representations comprises:
a plurality of nodes, each node representing an entity;
a plurality of edges, each edge connecting at least two nodes and defining a relationship between at least two entities represented by the at least two nodes; and
a plurality of properties, each property defining an attribute of one or more entities or one or more relationships;
and wherein each graph representation of the plurality of graph representations is derived from a plurality of extracted data points, each extracted data point comprising at least one of structured data, unstructured data, or time series data, wherein each extracted data point comprising unstructured data is assigned, using one or more machine learning models, to one or more of a plurality of categories based at least in part on a relationship between each extracted data point and at least one other extracted data point, wherein each extracted categorized data point comprising unstructured data is assigned to a cluster of a plurality of clusters based at least in part on a relationship of each extracted categorized data point to a characteristic of a source of the unstructured data, and wherein the assignment of categories and clusters to each extracted data point comprising unstructured data alters a structured data schema used to extract the structured data;
retrieving, by the computer system, the one or more graph representations of the plurality of graph representations;
scoring, by the computer system, the one or more graph representations based on a relevance of the one or more graph representations to the natural language query to identify at least one relevant graph representation;
generating, by the computer system, a response to the natural language query, wherein generating the response comprises providing the at least one relevant graph representation to a large language model (LLM), and wherein the response comprises:
natural language;
a visual representation of the graph traversal;
an indicator of one or more nodes of the plurality of nodes and one or more edges of the plurality of edges within the visual representation, wherein the one or more nodes and one or edges are determined to be relevant to the query; and
a structured citation comprising metadata indicating a source associated with the one or more nodes and one or more edges; and
displaying, by the user the user interface on the user device, the response to the natural language query.
3. A computer implemented method for query response, the method comprising:
receiving, via a user interface displayed at a user device, a query, wherein the query comprises natural language;
parsing and translating, by a computer system, the natural language query into a graph traversal or database query operation;
performing, by the computer system, the graph traversal or database query operation on a graph database to identify one or more graph representations, wherein the graph database comprises a plurality of graph representations defined by a graph schema, wherein each graph representation of the plurality of graph representations comprises:
a plurality of nodes, each node representing an entity;
a plurality of edges, each edge connecting at least two nodes and defining a relationship between at least two entities represented by the at least two nodes; and
a plurality of properties, each property defining an attribute of one or more entities or one or more relationships;
and wherein each graph representation of the plurality of graph representations is derived from a plurality of extracted data points, each extracted data point comprising at least one of structured data, unstructured data, or time series data, wherein each extracted data point comprising unstructured data is assigned, using one or more machine learning models, to one or more of a plurality of categories based at least in part on a relationship between each extracted data point and at least one other extracted data point, wherein each extracted categorized data point comprising unstructured data is assigned to a cluster of a plurality of clusters based at least in part on a relationship of each extracted categorized data point to a characteristic of a source of the unstructured data, and wherein the assignment of categories and clusters to each extracted data point comprising unstructured data alters a structured data schema used to extract the structured data;
retrieving, by the computer system, the one or more graph representations of the plurality of graph representations;
scoring, by the computer system, the one or more graph representations based on a relevance of the one or more graph representations to the natural language query to identify at least one relevant graph representation;
generating, by the computer system, a response to the natural language query, wherein generating the response comprises providing the at least one relevant graph representation to a large language model (LLM), and wherein the response comprises:
natural language;
a visual representation of the graph traversal;
an indicator of one or more nodes of the plurality of nodes and one or more edges of the plurality of edges within the visual representation, wherein the one or more nodes and one or edges are determined to be relevant to the query; and
a structured citation comprising metadata indicating a source associated with the one or more nodes and one or more edges; and
displaying, by the user the user interface on the user device, the response to the natural language query.
4. The method of claim 3 , wherein the graph database comprises a Neo4j, ArangoDB, or Amazon Neptune database, or wherein the graph database comprises a representation layer configured to interface with a relational database management system (RDBMS).
5. The method of claim 3 , wherein the unstructured data is ingested into the graph database by:
receiving, from a data source, one or more documents;
extracting text from the one or more documents;
extracting data from the extracted text using one or more large language models (LLMs), wherein the one or more LLMs are prompted to extract the data according to a predefined schema;
transforming the extracted data into a graph representation according to the graph schema;
storing the graph representation in the graph database.
6. The method of claim 5 , wherein the one or more documents comprise PDF files, plain text documents, rich text format (RTF) files, word documents, and/or image files containing text.
7. The method of claim 3 , wherein the structured data is ingested into the graph database by:
receiving, from a data source, one or more tabular data stores;
mapping the one or more tabular data stores to a graph representation using the graph schema, wherein the mapping comprises:
manually and/or automatically mapping a column of the one or more tabular data stores to a node of the plurality of nodes, an edge of the plurality of edges, or a property of the plurality of properties; and
storing the graph representation in the graph database.
8. The method of claim 3 , wherein the time series data is ingested into the graph database by:
receiving, from a data source, one or more temporal data sequences;
parsing and standardizing the one or more temporal data sequences to a common format;
detecting and correcting one or more missing data points within the one or temporal data sequences;
generating a graph representation of the one or more temporal data sequences using the graph schema;
storing the graph representation in the graph database and storing the one or more temporal data sequences in a storage location, wherein the graph representation comprises a reference to the storage location or a link to the one or more temporal data sequences.
9. The method of claim 8 , wherein the data source comprises a comma separated value (CSV) file with timestamp columns, a Javascript object notation (JSON) formatted time series, a specialized time series database, or a real-time data stream.
10. The method of claim 3 , wherein an edge of the plurality of edges defines a temporal relationship between the at least two entities represented by the at least two nodes, wherein the at least two nodes comprise time series nodes, and wherein the at least two entities comprise entities derived from the structured data or the unstructured data.
11. The method of claim 10 , wherein the temporal relationship is continuously updated based on one or more temporal data sequences received from a real-time data stream.
12. The method of claim 3 , wherein parsing and translating the natural language query into a graph traversal or database query operation comprises mapping the natural language query to a data category of the graph database and/or one or more nodes of the plurality of nodes.
13. The method of claim 3 , wherein parsing and translating the natural language query into a graph traversal or database query operation comprises:
generating an embedding for the natural language query using an embedding algorithm or a domain-specific embedding model; and
comparing the generated embedding to a pre-computed embedding associated with the unstructured data, structured data, or time series data using a similarity measure.
14. The method of claim 3 , wherein parsing and translating the natural language query into a graph traversal or database query operation comprises matching the natural language query to summaries of the structured data, unstructured data, and time series data based on embedding similarity.
15. The method of claim 3 , wherein scoring the one or more graph representations based on relevance comprises a combination of factors comprising category match strength, embedding similarity, and graph structural importance metrics.
16. The method of claim 3 , further comprising assigning, by the computer system, a confidence score to a plurality of parts of the generated response based on one or more factors including: data source reliability, consistency with multiple sources, and LLM confidence metrics.
17. The method of claim 3 , wherein providing the set of relevant graph representations to the LLM comprises:
formatting the set of relevant graph representations into a context document; and
providing the context document to the LLM.
18. The method of claim 3 , wherein each node representing an entity is linked to each node representing a same entity across the plurality of graph representations derived from at least one of the structured data, the unstructured data, and the time series data.
19. The method of claim 3 , wherein the graph schema is configured to automatically adjust, using machine learning, to include new types of nodes, edges, and/or properties.