Methods and systems for improved document processing and information retrieval
Disclosed are methods, systems, devices, apparatus, media, and other implementations that include a method for document processing (particularly for training of a machine learning question answering platform, and for ingestion of documents). The method includes obtaining a question dataset (e.g., either from public or private repositories of questions) comprising one or more source questions for document processing by a machine learning question-and-answer system that provides answer data in response to question data submitted by a user, modifying a source question from the question dataset to generate one or more augmented questions with equivalent semantic meanings as that of the source question, and processing a document with the one or more augmented questions.
1. A method for document processing, the method comprising:
obtaining a question dataset that comprises one or more source questions for document processing by a machine-learning question-and-answer system that provides answer data in response to question data submitted by a user,
analyzing the source question to determine a specificity level for the source question, wherein analyzing the source question comprises determining the source question is overly verbose based on a comparison of the determined specificity level to one or more specificity threshold values,
modifying the source question from the question dataset to generate one or more augmented questions that have equivalent semantic meanings as that of the source question,
processing a document with the one or more augmented questions,
wherein modifying the source question comprises: simplifying the source question, in response to a determination that the source question is overly verbose, to exclude one or more semantic elements of the source question to generate a terse question with an equivalent semantic meaning to the source question, and
adding the one or more augmented questions to an augmented question dataset.
2. The method of claim 1 , wherein processing the document with the one or more augmented questions comprises one or more of:
training the machine learning question-and-answer system using the document and the one or more augmented questions; or
ingesting the document with the one or more augmented questions, subsequent to completion of the training of the machine learning question-and-answer system, to generate an ingested document that is searchable by the machine learning question-and-answer system.
3. The method of claim 1 , wherein analyzing the source question comprises one or more of: determining a word count for the source question, determining intent associated with the source question, or classifying the source question using a machine-learning question specificity model.
4. The method of claim 1 , wherein simplifying the source question comprises excluding the one or more semantic elements of the source question according to one or more of: a term-weighing scheme to assign values for words appearing in the source question, or one or more natural language processing (NPL) rules applied to the source question.
5. The method of claim 4 , wherein the term-weighing scheme comprises a term frequency-inverse document frequency (TF-IDF) scheme, wherein simplifying the source question comprises:
computing weight values for the words appearing in the source question according to the (TF-IDF) scheme; and
removing one or more of the words appearing in the source question based on the computed weight values, and subject to the NPL rules.
6. The method of claim 1 , wherein modifying the source question comprises:
generating a question variant message comprising the source question and one or more required output characteristics for a resultant augmented question paraphrasing of the source question; and
providing the question variant message to a generative large language model system configured to generate a resultant augmented question based on the source question and the one or more required output characteristics.
7. The method of claim 1 , further comprising:
iteratively modifying the one or more augmented questions to generate additional sets of augmented questions with similar semantic meanings as that of a preceding set of augmented questions.
8. A method for document processing, the method comprising:
obtaining a question dataset that comprises one or more source questions for document processing by a machine-learning question-and-answer system that provides answer data in response to question data submitted by a user,
analyzing the source question to determine a specificity level for the source question, wherein analyzing the source question comprises determining the source question is overly verbose based on a comparison of the determined specificity level to one or more specificity threshold values,
modifying the source question from the question dataset to generate one or more augmented questions that have equivalent semantic meanings as that of the source question, and
processing a document with the one or more augmented questions,
wherein modifying the source question comprises: simplifying the source question, in response to a determination that the source question is overly verbose, to exclude one or more semantic elements of the source question to generate a terse question with an equivalent semantic meaning to the source question, and
wherein the source question includes unstructured keywords, and wherein modifying the source question comprises expanding the source question, in response to a determination that the source question is overly terse, to include structural components for the unstructured keywords.
9. The method of claim 8 , wherein expanding the source question comprises:
determining a statement of intent and/or other contextual information associated with the keywords of the source question; and
adding to the source question semantic structural components determined based on the statement of intent and/or other contextual information.
10. A method for document processing, the method comprising:
obtaining a question dataset comprising one or more source questions for document processing by a machine learning question-and-answer system that provides answer data in response to question data submitted by a user,
analyzing the source question to determine specificity level for the source question,
wherein analyzing the source question comprises determining that the source question is overly terse based on a comparison of the determined specificity level to one or more specificity threshold values,
modifying the source question from the question dataset to generate one or more augmented questions with equivalent semantic meanings as that of the source question;
processing a document with the one or more augmented questions,
wherein the source question includes unstructured keywords, and wherein modifying the source question comprises expanding the source question, in response to a determination that the source question is overly terse, to include structural components for the unstructured keywords, and
adding the one or more augmented questions to an augmented question dataset.
11. The method of claim 10 , wherein processing the document with the one or more augmented questions comprises one or more of:
training the machine learning question-and-answer system using the document and the one or more augmented questions; or
ingesting the document with the one or more augmented questions, subsequent to completion of the training of the machine learning question-and-answer system, to generate an ingested document that is searchable by the machine learning question-and-answer system.
12. The method of claim 10 , wherein analyzing the source question comprises one or more of: determining a word count for the source question, determining intent associated with the source question, or classifying the source question using a machine-learning question specificity model.
13. The method of claim 10 , wherein analyzing the source question comprises:
determining the source question is one of overly verbose or overly terse based on a comparison of the determined specificity level to one or more specificity threshold values.
14. The method of claim 10 , wherein modifying the source question comprises:
simplifying the source question, in response to a determination that the source question is overly verbose, to exclude one or more semantic elements of the source question to generate a terse question with an equivalent semantic meaning to the source question.
15. The method of claim 14 , wherein simplifying the source question comprises excluding the one or more semantic elements of the source question according to one or more of: a term-weighing scheme to assign values for words appearing in the source question, or one or more natural language processing (NPL) rules applied to the source question.
16. The method of claim 15 , wherein the term-weighing scheme comprises a term frequency-inverse document frequency (TF-IDF) scheme, wherein simplifying the source question comprises:
computing weight values for the words appearing in the source question according to the (TF-IDF) scheme; and
removing one or more of the words appearing in the source question based on the computed weight values, and subject to the NPL rules.
17. The method of claim 10 , wherein expanding the source question comprises:
determining a statement of intent and/or other contextual information associated with the keywords of the source question; and
adding to the source question semantic structural components determined based on the statement of intent and/or other contextual information.
18. The method of claim 10 , wherein modifying the source question comprises:
generating a question variant message comprising the source question and one or more required output characteristics for a resultant augmented question paraphrasing of the source question; and
providing the question variant message to a generative large language model system configured to generate a resultant augmented question based on the source question and the one or more required output characteristics.
19. The method of claim 10 , further comprising:
iteratively modifying the one or more augmented questions to generate additional sets of augmented questions with similar semantic meanings as that of a preceding set of augmented questions.