GROUNDING LARGE LANGUAGE MODELS USING REAL-TIME CONTENT FEEDS AND REFERENCE DATA
This disclosure describes a system and method, using one or more processors, for grounding large language models using real-time content feeds and reference data. The system is able to receive/generate queries, prompt an AI model and evaluate the answer given by the AI model. A hallucination score is generated according to the evaluation of the answer. According to the hallucination score, the AI model may be iteratively accessed to improve the truthfulness of the answer.
1 . A computing system:
one or more processors;
a reference database; and
one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising:
receiving an initial query;
processing the initial query with an artificial intelligence (AI) model to generate an output,
generating a plurality of metrics by comparing the output to the reference database according to a plurality of criterion,
determining a hallucination score according to an evaluation of the plurality of metrics.
2 . The computing system of claim 1 , wherein the operations comprise:
rewriting the initial query to generate a new query;
processing the new query with the AI model to generate a new output,
generating a plurality of new metrics by comparing the new output to the reference database according to the plurality of criterion,
determining a new hallucination score according to an evaluation of the plurality of new metrics.
3 . The computing system of claim 2 , wherein the new query is based on the hallucination score.
4 . The computing system of claim 2 , wherein the operations comprise:
iteratively performing the rewriting, the processing, the generating, and the determining until the hallucination score or the new hallucination score is acceptable.
5 . The computing system of claim 3 , wherein the hallucination score or the new hallucination score is acceptable as compared to a threshold.
6 . The computing system of claim 1 , wherein the operations comprise:
providing a user interface configured to receive a new query according to the hallucination score.
7 . The computing system of claim 6 , wherein the operations comprise:
processing the new query with the AI model to generate a new output,
generating a plurality of new metrics by comparing the new output to the reference database according to the plurality of criterion,
determining a new hallucination score according to an evaluation of the plurality of new metrics.
8 . The computing system of claim 7 , wherein the operations comprise:
iteratively performing the rewriting, the processing, the generating, and the determining until the hallucination score or the new hallucination score is acceptable.
9 . The computing system of claim 8 , wherein the hallucination score or the new hallucination score is acceptable as compared to a threshold.
10 . The computing system of claim 1 , wherein the operations comprise:
providing a user interface configured to allow a user to change the output according to the hallucination score.
11 . The computing system of claim 10 , wherein the operations comprise:
generating a new query according to the hallucination score and the changed output.
12 . The computing system of claim 11 , wherein the operations comprise:
processing the new query with the AI model to generate a new output,
generating a plurality of new metrics by comparing the new output to the reference database according to the plurality of criterion,
determining a new hallucination score according to an evaluation of the plurality of new metrics.
13 . The computing system of claim 12 , wherein the operations comprise:
iteratively performing the rewriting, the processing, the generating, and the determining until the hallucination score or the new hallucination score is acceptable.
14 . The computing system of claim 13 , wherein the hallucination score or the new hallucination score is acceptable as compared to a threshold.
15 . The computing system of claim 1 , wherein the AI model is a large language model (LLM).
16 . The computing system of claim 1 , wherein the reference database comprises is a real-time content feed.
17 . A computing system:
one or more processors;
a first reference database; and
one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising:
receiving an initial query;
processing the initial query with an artificial intelligence (AI) model to generate a first output,
inputting the first output into a system which contains a large language model (LLM) to generate a second output,
generating a plurality of metrics by comparing the second output to a second reference database according to a plurality of criterion,
determining a hallucination score according to an evaluation of the plurality of metrics.
18 . The computing system of claim 1 , wherein the operations comprise:
rewriting the initial query to generate a new query;
processing the new query with the AI model to generate a third output,
inputting the third output into the system which contains an LLM to generate a fourth output,
generating a plurality of new metrics by comparing the fourth output a third reference database according to the plurality of criterion, and
determining a new hallucination score according to an evaluation of the plurality of new metrics.