Systems and methods for generating a contextually and conversationally correct response to a query
The present disclosure relates to systems and methods for generating contextually, grammatically, and conversationally correct answers to input questions. Embodiments provide for linguistic and syntactic structure analysis of a submitted question in order to determine whether the submitted question may be answered by at least one headnote. The question is then further analyzed to determine more details about the intent and context of the question. A federated search process, based on the linguistic and syntactic structure analysis, and the additional analysis of the question is used to identify candidate question-answer pairs from a corpus of previously created headnotes. Machine learning models are used to analyze the candidate question-answer pairs, additional rules are applied to rank the candidate answers, and dynamic thresholds are applied to identify the best potential answers to provide to a user as a response to the submitted question.
1. A method comprising:
receiving a query from a user terminal, the query including a question having a linguistic and syntactic structure;
analyzing the linguistic and syntactic structure of the question to determine at least a context of the question;
pre-processing the received question prior to the analyzing, wherein the pre-processing includes determining whether the question is of a type that can be answered by a headnote,
generating at least one search query based on the analyzing the linguistic and syntactic structure of the question;
causing the at least one search query to be executed on at least one data store;
obtaining a plurality of candidate answers in response to the execution of the search query, wherein each candidate answer of the plurality of candidate answers is a headnote;
obtaining a linguistic and syntactic analysis of each candidate answer of the plurality of candidate answers, wherein the question is paired with each candidate answer of the plurality of candidate answers to form a plurality of question-answer pairs;
extracting at least one feature for each question-answer pair of the plurality of question-answer pairs;
feeding the extracted at least one feature for each question-answer pair into a ranking model;
scoring, by the ranking model, for each feature of the at least one feature, each question-answer pair, wherein a score for a particular question-answer pair resulting from the scoring represents a probability that the particular candidate answer of the particular question-answer pair is a correct answer to the question;
ranking the candidate answers of the plurality of candidate answers based on the score of each candidate answer; and
providing at least one of the ranked candidate answers as answer to the question in the query.
2. The method of claim 1 , wherein the analyzing the linguistic and syntactic structure of the question includes one of: natural language processing, entity recognition, frame classification, key number classification, and embeddings analysis.
3. The method of claim 2 , wherein the context of the question includes the intent of the question, and wherein the frame classification includes classifying the question into at least one frame category associated with the intent of the question.
4. The method of claim 2 , wherein the at least one frame category is one of: Admissibility, Availability of Damages or Remedy, Burden of Proof, Construction of Instruments, Court Authority, Elements, Factors, and Tests, Accrual of Statute of Limitations, Tolling of Statute of Limitations, Duration of Statute of Limitations, Standard of Review, Enforceability of Contracts, and Others.
5. The method of claim 1 , wherein the obtaining the linguistic and syntactic analysis of each candidate answer of the plurality of candidate answers includes analyzing the linguistic and syntactic structure of each candidate answer subsequent to the obtaining the plurality of candidate answers in response to the execution of the search query.
6. The method of claim 1 , wherein the obtaining the linguistic and syntactic analysis of each candidate answer of the plurality of candidate answers includes analyzing the linguistic and syntactic structure of each candidate answer prior to the obtaining the plurality of candidate answers in response to the execution of the search query, wherein the linguistic and syntactic analysis of each candidate answer is obtained from a database.
7. The method of claim 1 , further comprising post-processing the ranked candidate answers, wherein the post-processing includes one of:
applying constraint rules to the ranked candidate answers, to ensure that the candidate answers contain required elements, wherein a particular ranked candidate answer is eliminated as a candidate answer based on a constraint rule being met; and
applying weighting rules to one of penalize and boost a ranked candidate answer.
8. The method of claim 1 , wherein the providing at least one of the ranked candidate answers as answer to the question in the query includes performing a threshold determination to the at least one of the ranked candidate answers to determine whether the at least one of the ranked candidate answers will be provided as an answer to the question based on a threshold.
9. The method of claim 8 , wherein the threshold is a threshold value of a probability that the at least one of the ranked candidate answers is a correct answer, wherein the at least one of the ranked candidate answers is determined to be provided as an answer to the question when a probability score of the at least one of the ranked candidate answers exceeds the threshold value.
10. The method of claim 1 , wherein the extracted at least one feature includes at least one of: a linguistic similarity feature, a concept coordination feature, a topicality feature, an abstract/concrete classification feature, and a key number scoring function feature.
11. The method of claim 1 , wherein the search query includes one of: a natural language search query, a more-like-this search query, and a semantic search query.
12. The method of claim 11 , wherein the analyzing the linguistic and syntactic structure of the question includes determining a frame into which the question is classified, wherein the semantic search query includes:
obtaining a frame-specific search template, the frame-specific search template associated with the frame of the question, and wherein the frame-specific search template defines at least one placeholder corresponding to an element of the frame of the question;
identifying an entity in the question corresponding to the element of the frame corresponding to the at least one placeholder; and
replacing the placeholder in the frame-specific search template with the identified entity to generate a complete frame-specific search template, wherein the generating at least one search query is based on the complete frame-specific search template.
13. A system comprising:
a question/answer processor configured to:
receive a query from a user terminal, the query including a question having a linguistic and syntactic structure;
analyze the linguistic and syntactic structure of the question to determine at least a context of the question; and
pre-process the received question prior to analyzing the linguistic and syntactic structure of the question, wherein pre-processing includes determining whether the question is of a type that can be answered by a headnote;
a query generator configured to:
generate at least one search query based on the analyzing the linguistic and syntactic structure of the question;
cause the at least one search query to be executed on at least one data store; and
obtain a plurality of candidate answers in response to the execution of the search query, wherein each candidate answer of the plurality of candidate answers is a headnote, wherein the question/answer processor is further configured to obtain a linguistic and syntactic analysis of each candidate answer of the plurality of candidate answers, and to pair the question with each candidate answer of the plurality of candidate answers to form a plurality of question-answer pairs;
a feature extractor configured to:
extract at least one feature for each question-answer pair of the plurality of question-answer pairs; and
feed the extracted at least one feature for each question-answer pair into a candidate ranker;
the candidate ranker configured to:
score, using a ranking model, for each feature of the at least one feature, each question-answer pair, wherein a score for a particular question-answer pair resulting from the scoring represents a probability that the particular candidate answer of the particular question-answer pair is a correct answer to the question; and
rank the candidate answers of the plurality of candidate answers based on the score of each candidate answer; and
an answer detector configured to provide at least one of the ranked candidate answers as answer to the question in the query.
14. The system of claim 13 , wherein the configuration of the question/answer processor to analyze the linguistic and syntactic structure of the question includes configuration of the question/answer processor to perform one of: natural language processing, entity recognition, frame classification, key number classification, and embeddings analysis.
15. The system of claim 13 , wherein the configuration of the question/answer processor to obtain the linguistic and syntactic analysis of each candidate answer of the plurality of candidate answers includes configuration of the question/answer processor to one of:
analyze the linguistic and syntactic structure of each candidate answer subsequent to obtaining the plurality of candidate answers in response to the execution of the search query; and
analyze the linguistic and syntactic structure of each candidate answer prior to obtaining the plurality of candidate answers in response to the execution of the search query, wherein the linguistic and syntactic analysis of each candidate answer is obtained from a database.
16. The system of claim 13 , wherein the configuration of the answer detector to provide at least one of the ranked candidate answers as answer to the question in the query includes configuration of the answer detector to perform a threshold determination to the at least one of the ranked candidate answers to determine whether the at least one of the ranked candidate answers will be provided as an answer to the question based on a threshold, wherein the threshold is a threshold value of a probability that the at least one of the ranked candidate answers is a correct answer, wherein the at least one of the ranked candidate answers is determined to be provided as an answer to the question when a probability score of the at least one of the ranked candidate answers exceeds the threshold value.
17. The system of claim 13 , wherein the search query includes one of: a natural language search query, a more-like-this search query, and a semantic search query.
18. The system of claim 17 , wherein the configuration of the question/answer processor to analyze the linguistic and syntactic structure of the question includes configuration to determine a frame into which the question is classified, wherein the semantic search query includes:
obtaining a frame-specific search template, the frame-specific search template associated with the frame of the question, and wherein the frame-specific search template defines at least one placeholder corresponding to an element of the frame of the question;
identifying an entity in the question corresponding to the element of the frame corresponding to the at least one placeholder; and
replacing the placeholder in the frame-specific search template with the identified entity to generate a complete frame-specific search template, wherein the generating at least one search query is based on the complete frame-specific search template.
19. A computer-based tool including non-transitory computer readable media having stored thereon computer code which, when executed by a processor, causes a computing device to perform operations comprising:
receiving a query from a user terminal, the query including a question having a linguistic and syntactic structure;
analyzing the linguistic and syntactic structure of the question to determine at least a context of the question;
pre-processing the received question prior to the analyzing, wherein the pre-processing includes determining whether the question is of a type that can be answered by a headnote,
generating at least one search query based on the analyzing the linguistic and syntactic structure of the question;
causing the at least one search query to be executed on at least one data store;
obtaining a plurality of candidate answers in response to the execution of the search query, wherein each candidate answer of the plurality of candidate answers is a headnote;
obtaining a linguistic and syntactic analysis of each candidate answer of the plurality of candidate answers, wherein the question is paired with each candidate answer of the plurality of candidate answers to form a plurality of question-answer pairs;
extracting at least one feature for each question-answer pair of the plurality of question-answer pairs;
feeding the extracted at least one feature for each question-answer pair into a ranking model;
scoring, by the ranking model, for each feature of the at least one feature, each question-answer pair, wherein a score for a particular question-answer pair resulting from the scoring represents a probability that the particular candidate answer of the particular question-answer pair is a correct answer to the question;
ranking the candidate answers of the plurality of candidate answers based on the score of each candidate answer; and
providing at least one of the ranked candidate answers as answer to the question in the query.