Corpus search improvements using term normalization
System and computer program product to perform an operation for query processing based on normalized search terms. The operation begins by, responsive to receiving a query, generating a normalized search term for a concept in the query based on a first language model, of a plurality of language models each having a predefined association with a respective concept. The operation then modifies the query to include the normalized search term, and executes the modified query against an indexed corpus of evidence including a first item of evidence. The operation then, upon determining that the first item of evidence includes the normalized search term, returns the first item of evidence as responsive to the query.
1. A method of query processing based on normalized search terms, the method comprising:
responsive to receiving a query, generating a normalized search term for a concept in the query based on a first language model, of a plurality of language models each having a predefined association with a respective concept;
modifying the query to include the normalized search term;
executing the modified query against the indexed corpus of evidence, where the corpus of evidence is indexed based on the plurality of language models to include a set of normalized terms for each respective item of evidence in the corpus, wherein the indexed corpus of evidence includes a first item of evidence used to support a first candidate answer, of a plurality of candidate answers; and
upon determining that the set of normalized terms for the first item of evidence includes the normalized search term, returning the first candidate answer as responsive to the query.
2. The method of claim 1 , further comprising generating the plurality of language models, wherein each language model of the plurality corresponds to a respective concept, wherein generating the respective language models comprises:
identifying at least one key term for the respective concept; and
generating a respective normalized search term representing the respective at least one key term.
3. The method of claim 2 , wherein the normalized search term is based on at least one of: (i) at least one variant of the at least one key term, and (ii) a context of the at least one key term.
4. The method of claim 3 , wherein the set of normalized terms comprises two or more normalized terms, wherein the corpus of evidence is further indexed by:
associating the set of normalized terms with the respective item of evidence; and
storing the association.
5. The method of claim 1 , wherein the concept is separately expressed by each of a plurality of variants.
6. The method of claim 5 , wherein the query is received from a requesting entity, wherein the query is processed without requiring the requesting entity to specify any of the variants other than a first variant included in the query.
7. The method of claim 1 , wherein the corpus of evidence is a closed corpus.