TECHNIQUES FOR DETECTING HALLUCINATION IN MACHINE-GENERATED RESPONSES
Systems and methods may provide techniques for detecting hallucination in machine-generated responses. A computer-implemented method can include accessing text data. The text data can include one or more machine-generated responses that are supplemented by outputs generated by a retrieval-augmentation generation (RAG) system. In some instances, the outputs are associated with the prompt associated with a user. The computer-implemented method can also include applying one or more hallucination-detection models to the text data to generate a set of classification labels. A classification label can indicate whether a corresponding machine-generated response of the one or more machine-generated responses contradicts at least part of the knowledge base accessed by the RAG system. The computer-implemented method can also include generating annotated text data that includes the one or more machine-generated responses annotated with corresponding classification labels of the set of classification labels. The computer-implemented method can also include outputting the annotated text data.
1 . A computer-implemented method comprising:
accessing text data, wherein the text data includes one or more machine-generated responses that are supplemented by outputs generated by a retrieval-augmentation generation (RAG) system, and wherein the one or more machine-generated responses are associated with a prompt associated with a user;
applying one or more hallucination-detection models to the text data to generate a set of classification labels, wherein a classification label of the set of classification labels indicates whether a corresponding machine-generated response of the one or more machine-generated responses contradicts at least part of a knowledge base accessed by the RAG system, and wherein the one or more hallucination-detection models were trained using a training dataset that includes previous machine-generated responses annotated with the set of classification labels;
generating annotated text data that includes the one or more machine-generated responses annotated with corresponding classification labels of the set of classification labels; and
outputting the annotated text data.
2 . The computer-implemented method of claim 1 , wherein generating the outputs includes:
encoding the prompt into one or more embeddings, wherein when the one or more embeddings are entered in a database, the RAG system uses prompt results outputted from the database to supplement the one or more machine-generated responses.
3 . The computer-implemented method of claim 1 , wherein the knowledge base includes domain-specific information, wherein the domain-specific information is associated with a particular domain.
4 . The computer-implemented method of claim 1 , wherein the set of classification labels includes a no-info classification label indicating that the corresponding machine-generated response includes non-verifiable information, a supported classification label indicating that the corresponding machine-generated response is supported by the knowledge base, and an unsupported classification label indicating that the corresponding machine-generated response contradicts the at least part of the knowledge base.
5 . The computer-implemented method of claim 1 , wherein the one or more hallucination-detection models include a pretrained Decoding-enhanced Bidirectional Encoder Representations from Transformers with Disentangled attention (DeBERTa) model.
6 . The computer-implemented method of claim 1 , wherein the one or more hallucination-detection models include a pretrained large-language model (LLM).
7 . The computer-implemented method of claim 1 , wherein applying the one or more hallucination-detection models to the text data includes:
applying a first hallucination-detection model of the one or more hallucination-detection models to the text data to generate a verifiable classification label indicating that the corresponding machine-generated response includes information verifiable from the knowledge base; and
applying a second hallucination-detection model of the one or more hallucination-detection models to the corresponding machine-generated response to generate the classification label indicating whether the corresponding machine-generated response contradicts at least part of the knowledge base.
8 . The computer-implemented method of claim 1 , wherein applying the one or more hallucination-detection models to the text data includes:
applying a first hallucination-detection model of the one or more hallucination-detection models to the text data to generate a no-info classification label indicating that the corresponding machine-generated response includes non-verifiable information.
9 . The computer-implemented method of claim 1 , wherein outputting the annotated text data includes displaying in real-time the annotated text data on a graphical user interface, as messages are exchanged between the user and an agent during an instant-chat session.
10 . A system comprising:
one or more processors; and
memory storing thereon instructions that, as a result of being executed by the one or more processors, cause the system to perform operations comprising:
accessing text data, wherein the text data includes one or more machine-generated responses that are supplemented by outputs generated by a retrieval-augmentation generation (RAG) system, and wherein the one or more machine-generated responses are associated with a prompt associated with a user;
applying one or more hallucination-detection models to the text data to generate a set of classification labels, wherein a classification label of the set of classification labels indicates whether a corresponding machine-generated response of the one or more machine-generated responses contradicts at least part of a knowledge base accessed by the RAG system, and wherein the one or more hallucination-detection models were trained using a training dataset that includes previous machine-generated responses annotated with the set of classification labels;
generating annotated text data that includes the one or more machine-generated responses annotated with corresponding classification labels of the set of classification labels; and
outputting the annotated text data.
11 . The system of claim 10 , wherein generating the outputs includes:
encoding the prompt into one or more embeddings, wherein when the one or more embeddings are entered in a database, the RAG system uses prompt results outputted from the database to supplement the one or more machine-generated responses.
12 . The system of claim 10 , wherein the knowledge base includes domain-specific information, wherein the domain-specific information is associated with a particular domain.
13 . The system of claim 10 , wherein the set of classification labels includes a no-info classification label indicating that the corresponding machine-generated response includes non-verifiable information, a supported classification label indicating that the corresponding machine-generated response is supported by the knowledge base, and an unsupported classification label indicating that the corresponding machine-generated response contradicts the at least part of the knowledge base.
14 . The system of claim 10 , wherein the one or more hallucination-detection models include a pretrained Decoding-enhanced Bidirectional Encoder Representations from Transformers with Disentangled attention (DeBERTa) model.
15 . The system of claim 10 , wherein the one or more hallucination-detection models include a pretrained large-language model (LLM).
16 . The system of claim 10 , wherein applying the one or more hallucination-detection models to the text data includes:
applying a first hallucination-detection model of the one or more hallucination-detection models to the text data to generate a verifiable classification label indicating that the corresponding machine-generated response includes information verifiable from the knowledge base; and
applying a second hallucination-detection model of the one or more hallucination-detection models to the corresponding machine-generated response to generate the classification label indicating whether the corresponding machine-generated response contradicts at least part of the knowledge base.
17 . The system of claim 10 , wherein applying the one or more hallucination-detection models to the text data includes:
applying a first hallucination-detection model of the one or more hallucination-detection models to the text data to generate a no-info classification label indicating that the corresponding machine-generated response includes non-verifiable information.
18 . The system of claim 10 , wherein outputting the annotated text data includes displaying in real-time the annotated text data on a graphical user interface, as messages are exchanged between the user and an agent during an instant-chat session.
19 . A non-transitory, computer-readable storage medium storing thereon executable instructions that, as a result of being executed by one or more processors of a computer system, cause the computer system to perform operations comprising:
accessing text data, wherein the text data includes one or more machine-generated responses that are supplemented by outputs generated by a retrieval-augmentation generation (RAG) system, and wherein the one or more machine-generated responses are associated with a prompt associated with a user;
applying one or more hallucination-detection models to the text data to generate a set of classification labels, wherein a classification label of the set of classification labels indicates whether a corresponding machine-generated response of the one or more machine-generated responses contradicts at least part of a knowledge base accessed by the RAG system, and wherein the one or more hallucination-detection models were trained using a training dataset that includes previous machine-generated responses annotated with the set of classification labels;
generating annotated text data that includes the one or more machine-generated responses annotated with corresponding classification labels of the set of classification labels; and
outputting the annotated text data.
20 . The non-transitory, computer-readable storage medium of claim 19 , wherein generating the outputs includes:
encoding the prompt into one or more embeddings, wherein when the one or more embeddings are entered in a database, the RAG system uses prompt results outputted from the database to supplement the one or more machine-generated responses.
21 . The non-transitory, computer-readable storage medium of claim 19 , wherein the knowledge base includes domain-specific information, wherein the domain-specific information is associated with a particular domain.
22 . The non-transitory, computer-readable storage medium of claim 19 , wherein the set of classification labels includes a no-info classification label indicating that the corresponding machine-generated response includes non-verifiable information, a supported classification label indicating that the corresponding machine-generated response is supported by the knowledge base, and an unsupported classification label indicating that the corresponding machine-generated response contradicts the at least part of the knowledge base.
23 . The non-transitory, computer-readable storage medium of claim 19 , wherein the one or more hallucination-detection models include a pretrained Decoding-enhanced Bidirectional Encoder Representations from Transformers with Disentangled attention (DeBERTa) model.
24 . The non-transitory, computer-readable storage medium of claim 19 , wherein the one or more hallucination-detection models include a pretrained large-language model (LLM).
25 . The non-transitory, computer-readable storage medium of claim 19 , wherein applying the one or more hallucination-detection models to the text data includes:
applying a first hallucination-detection model of the one or more hallucination-detection models to the text data to generate a verifiable classification label indicating that the corresponding machine-generated response includes information verifiable from the knowledge base; and
applying a second hallucination-detection model of the one or more hallucination-detection models to the corresponding machine-generated response to generate the classification label indicating whether the corresponding machine-generated response contradicts at least part of the knowledge base.
26 . The non-transitory, computer-readable storage medium of claim 19 , wherein applying the one or more hallucination-detection models to the text data includes:
applying a first hallucination-detection model of the one or more hallucination-detection models to the text data to generate a no-info classification label indicating that the corresponding machine-generated response includes non-verifiable information.
27 . The non-transitory, computer-readable storage medium of claim 19 , wherein outputting the annotated text data includes displaying in real-time the annotated text data on a graphical user interface, as messages are exchanged between the user and an agent during an instant-chat session.