Automated citations and assessment for automatically generated text

1. A method comprising:

parsing, by a processor, a generated text to identify statements included within a generated text;

querying, by the processor, a remote data source to identify sources for each statement in the statements;

determining, by the processor, trustworthiness values for each statement, a trustworthiness value for a given statement determined by computing trustworthiness labels for each source corresponding to a given statement:

generating, by the processor, a label for the generated text based on an aggregated trustworthiness of each of the statements;

displaying, by the processor, the generated text and the label within a user interface displayed to a user;

displaying statement labels for each of the statements, the statement labels representing trustworthiness of each of the statements; and

detecting a selection of a given statement label and displaying a corresponding list of sources.

2. The method of claim 1 , further comprising generating the generated text by inserting a prompt into a large language model (LLM).

3. The method of claim 1 , wherein identifying the statements included within the generated text includes:

parsing the generated text into a set of sentences;

filtering the set of sentences to identify a second set of sentences that includes informational content; and

using the second set of sentences as the statements.

4. The method of claim 3 , wherein identifying the statements included within the generated text includes performing co-reference resolution on the generated text prior to parsing the generated text.

5. The method of claim 1 , wherein identifying a given source in the sources comprises using a corresponding statement as a search query and querying a search engine to obtain a list of search results, the list of search results comprising the sources.

6. The method of claim 1 , wherein determining trustworthiness values of each statement comprises:

classifying a trustworthiness of each source in sources corresponding to the given statement;

scoring and sorting the sources corresponding to the given statement;

selecting a top subset of the sources; and

determining a given trustworthiness value of the given statement based on classified trustworthiness values of each source in the top subset of the sources.

7. The method of claim 6 , wherein selecting a top subset of the sources comprises performing an entailment analysis using the given statement and a respective source.

8. A non-transitory computer-readable storage medium for tangibly storing computer program instructions capable of being executed by a computer processor, the computer program instructions defining steps of:

parsing a generated text to identify statements included within a generated text;

querying a remote data source to identify sources for each statement in the statements;

determining trustworthiness values for each statement, a trustworthiness value for a given statement determined by computing trustworthiness labels for each source corresponding to a given statement:

generating a label for the generated text based on an aggregated trustworthiness of each of the statements;

displaying the generated text and the label within a user interface displayed to a user;

displaying statement labels for each of the statements, the statement labels representing trustworthiness of each of the statements; and

detecting a selection of a given statement label and displaying a corresponding list of sources.

9. The non-transitory computer-readable storage medium of claim 8 , the steps further comprising generating the generated text by inserting a prompt into a large language model (LLM).

10. The non-transitory computer-readable storage medium of claim 8 , wherein identifying the statements included within the generated text includes:

parsing the generated text into a set of sentences;

filtering the set of sentences to identify a second set of sentences that includes informational content; and

using the second set of sentences as the statements.

11. The non-transitory computer-readable storage medium of claim 10 , wherein identifying the statements included within the generated text includes performing co-reference resolution on the generated text prior to parsing the generated text.

12. The non-transitory computer-readable storage medium of claim 8 , wherein identifying a given source in the sources comprises using a corresponding statement as a search query and querying a search engine to obtain a list of search results, the list of search results comprising the sources.

13. The non-transitory computer-readable storage medium of claim 8 , wherein determining trustworthiness values of each statement comprises:

classifying a trustworthiness of each source in sources corresponding to the given statement;

scoring and sorting the sources corresponding to the given statement;

selecting a top subset of the sources; and

determining a given trustworthiness value of the given statement based on classified trustworthiness values of each source in the top subset of the sources.

14. The non-transitory computer-readable storage medium of claim 13 , wherein selecting a top subset of the sources comprises performing an entailment analysis using the given statement and a respective source.

15. A device comprising:

a processor; and

a storage medium for tangibly storing thereon logic for execution by the processor, the logic comprising instructions for:

parsing a generated text to identify statements included within a generated text;

querying a remote data source to identify sources for each statement in the statements;

determining trustworthiness values for each statement, a trustworthiness value for a given statement determined by computing trustworthiness labels for each source corresponding to a given statement:

generating a label for the generated text based on an aggregated trustworthiness of each of the statements;

displaying the generated text and the label within a user interface displayed to a user;

displaying statement labels for each of the statements, the statement labels representing trustworthiness of each of the statements; and

detecting a selection of a given statement label and displaying a corresponding list of sources.

16. The device of claim 15 , wherein identifying the statements included within the generated text includes:

parsing the generated text into a set of sentences;

filtering the set of sentences to identify a second set of sentences that includes informational content; and

using the second set of sentences as the statements.

17. The device of claim 15 , wherein identifying a given source in the sources comprises using a corresponding statement as a search query and querying a search engine to obtain a list of search results, the list of search results comprising the sources.

18. The device of claim 15 , wherein determining trustworthiness values of each statement comprises:

classifying a trustworthiness of each source in sources corresponding to the given statement;

scoring and sorting the sources corresponding to the given statement;

selecting a top subset of the sources by performing an entailment analysis using the given statement and a respective source; and

determining a given trustworthiness value of the given statement based on classified trustworthiness values of each source in the top subset of the sources.