HALLUCINATION PREVENTION FOR NATURAL LANGUAGE INSIGHTS
Methods and systems are provided for hallucination prevention for natural language insights. In embodiments described herein, a template-based insight with a set of facts is generated by a template-based insights engine. The set of facts are generated from a set of data and the template-based insight is generated based on a text template. A natural language insight is generated from the template-based insight using a language model. If a single fact of the template-based insight is missing from the natural language insight, the single missing fact is an integer and a remaining integer of the natural language insight is within a threshold edit distance of the integer, the hallucination of the natural language insight is corrected by replacing the remaining integer with the single missing fact.
1 . A computer-implemented method comprising:
accessing, by a template-based insights engine, a template-based insight with a set of facts;
prompting a language model to generate a natural language insight based on the template-based insight;
before causing display of the natural language insight:
determining only a single fact from the set of facts of the template-based insight to be missing from the natural language insight; and
based on only the single fact from the set of facts of the template-based insight missing from the natural language insight:
identifying the single fact as an integer;
identifying whether a remaining fact of the natural language insight is a different integer; and
based on the different integer being less than a threshold edit distance from the integer, automatically replacing, by the hallucination gatekeeper engine, the different integer of the natural language insight with the integer; and
causing display of the natural language insight with the integer as replaced.
2 . The computer-implemented method of claim 1 , further comprising:
determining only the single fact from the set of facts of the template-based insight to be missing from the natural language insight based on searching the natural language insight for each fact in the set of facts of the template-based insight in order of a corresponding string length of each fact.
3 . The computer-implemented method of claim 1 , further comprising:
before determining only the single fact from the set of facts of the template-based insight to be missing from the natural language insight, determining that a number of facts with integers in the natural language insight is equal to a number of facts with integers in the set of facts of the template-based insight.
4 . The computer-implemented method of claim 1 , wherein the set of facts include integers and strings generated from the set of data.
5 . The computer-implemented method of claim 1 , wherein the threshold edit distance is one.
6 . The computer-implemented method of claim 1 , further comprising:
causing fine-tuning of a different language model to generate natural language insights from template-based insights using the template-based insight and the natural language insight with the integer as replaced.
7 . The computer-implemented method of claim 1 , further comprising:
determining, by the hallucination gatekeeper engine, that a corresponding set of facts in the natural language insight with the integer as replaced matches the set of facts of the template-based insight; and
automatically causing display of the new natural language insight with the integer as replaced further based on the corresponding set of facts in the new natural language insight matching the set of facts of the template-based insight.
8 . A non-transitory computer-readable medium storing executable instructions, which when executed by a processing device, cause the processing device to perform operations comprising:
generating, by a template-based insights engine, a template-based insight with a set of facts generated from a set of data, the template-based insight being generated in text form based on a text template;
generating, by a language model, a natural language insight from the template-based insight;
before causing display of the natural language insight:
searching the natural language insight for each fact in the set of facts of the template-based insight in order of a corresponding string length of each fact;
based on the search, determining only a single fact from the set of facts of the template-based insight to be missing from the natural language insight; and
based on only the single fact from the set of facts of the template-based insight missing from the natural language insight:
identifying the single fact as an integer;
identifying whether a remaining fact of the natural language insight is a different integer; and
based on the different integer being less than a threshold edit distance from the integer, automatically replacing, by the hallucination gatekeeper engine, the different integer of the natural language insight with the integer; and
causing display of the natural language insight with the integer as replaced.
9 . The non-transitory computer-readable medium of claim 8 , the operations further comprising:
before determining only the single fact from the set of facts of the template-based insight to be missing from the natural language insight, determining that a number of facts with integers in the natural language insight is equal to a number of facts with integers in the set of facts of the template-based insight.
10 . The non-transitory computer-readable medium of claim 8 , wherein the set of facts include integers and strings generated from the set of data.
11 . The non-transitory computer-readable medium of claim 8 , wherein the threshold edit distance is one.
12 . The non-transitory computer-readable medium of claim 8 , the operations further comprising:
causing fine-tuning of a different language model to generate natural language insights from template-based insights using the template-based insight and the natural language insight with the integer as replaced.
13 . The non-transitory computer-readable medium of claim 8 , the operations further comprising:
determining, by the hallucination gatekeeper engine, that a corresponding set of facts in the natural language insight with the integer as replaced matches the set of facts of the template-based insight; and
automatically causing display of the new natural language insight with the integer as replaced further based on the corresponding set of facts in the new natural language insight matching the set of facts of the template-based insight.
14 . A computing system comprising:
a processor; and
a non-transitory computer-readable medium having stored thereon instructions that when executed by the processor, cause the processor to perform operations including:
accessing, by a template-based insights engine, a template-based insight with a set of facts generated from a set of data, the template-based insight being generated in text form based on a text template;
prompting a language model to generate a natural language insight based on the template-based insight;
before causing display of the natural language insight:
determining only a single fact from the set of facts of the template-based insight to be missing from the natural language insight; and
based on only the single fact from the set of facts of the template-based insight missing from the natural language insight:
identifying the single fact as an integer;
identifying whether a remaining fact of the natural language insight is a different integer; and
based on the different integer being less than a threshold edit distance from the integer, automatically replacing, by the hallucination gatekeeper engine, the different integer of the natural language insight with the integer; and
causing display of the natural language insight with the integer as replaced.
15 . The system of claim 14 , wherein the instructions that when executed by the processor, cause the processor to perform operations further including:
determining only the single fact from the set of facts of the template-based insight to be missing from the natural language insight based on searching the natural language insight for each fact in the set of facts of the template-based insight in order of a corresponding string length of each fact.
16 . The system of claim 14 , wherein the instructions that when executed by the processor, cause the processor to perform operations further including:
before determining only the single fact from the set of facts of the template-based insight to be missing from the natural language insight, determining that a number of facts with integers in the natural language insight is equal to a number of facts with integers in the set of facts of the template-based insight.
17 . The system of claim 14 , wherein the set of facts include integers and strings generated from the set of data.
18 . The system of claim 14 , wherein the threshold edit distance is one.
19 . The system of claim 14 , wherein the instructions that when executed by the processor, cause the processor to perform operations further including:
causing fine-tuning of a different language model to generate natural language insights from template-based insights using the template-based insight and the natural language insight with the integer as replaced.
20 . The system of claim 14 , wherein the instructions that when executed by the processor, cause the processor to perform operations further including:
determining, by the hallucination gatekeeper engine, that a corresponding set of facts in the natural language insight with the integer as replaced matches the set of facts of the template-based insight; and
automatically causing display of the new natural language insight with the integer as replaced further based on the corresponding set of facts in the new natural language insight matching the set of facts of the template-based insight.