Augmenting queries with synonyms from synonyms map
Methods, systems, and apparatus, including computer program products, operable to perform operations including receiving through a user interface with an interface language a search query having query terms; using the interface language to select one or more mappings and using the selected mappings to simplify each query term; and applying each simplified query term to a synonyms map to identify possible synonyms with which to augment the search query. In alternative embodiments, the operations include generating a synonyms map from a corpus of documents; where the synonyms map maps each of multiple keys to one or more corresponding variants, where each variant is associated with one or more of document languages. In alternative embodiments, the operations include generating a synonyms map from documents by applying document language-dependent mappings to words in the documents to generate keys for the map.
1. A computer implemented method, comprising:
receiving, in a system comprising one or more computers, by way of a user interface, a search query comprising one or more query terms, the user interface having an interface language that is a natural language in which the user interface presents information;
based on the interface language, selecting one or more mappings and using the selected one or more mappings to simplify each query term to a corresponding simplified query term;
for each of one or more of the simplified query terms, identifying one or more synonyms for the simplified query term based on (i) a respective probability that each term in the search query is in a respective language based on counts of occurrences of the term in documents in each of a plurality of languages, and (ii) a frequency of the synonym in the documents in each of the plurality of languages, wherein the probability that the term in the search query is in a respective language is calculated based at least in part on a ratio between the count and a number of the plurality of languages; and
augmenting the search query with one or more of the identified synonyms.
2. The method of claim 1 wherein the calculation of the probability includes application of a smoothing factor to reduce noise.
3. The computer implemented method of claim 2 wherein the smoothing factor is based on counts of occurrences of each term and the number of languages.
4. The method of claim 1 wherein augmenting the query further comprises appending the one or more identified synonyms for a simplified query term to the search query wherein the one or more identified synonyms are appended as a conjunction with the simplified query term.
5. The computer implemented method of claim 1 , wherein the one or more mappings comprise one or more character conversion maps.
6. The computer implemented method of claim 5 , wherein the one or more character conversion maps comprise conversion maps for mapping words that include collapsible digraphs or language specific conversion maps.
7. A computer program product, encoded on a machine readable storage device or a machine-readable storage substrate, operable to cause data processing apparatus to perform operations comprising:
receiving, in a system comprising one or more computers, by way of a user interface, a search query comprising one or more query terms, the user interface having an interface language that is a natural language in which the user interface presents information;
based on the interface language, selecting one or more mappings and using the selected one or more mappings to simplify each query term to a corresponding simplified query term;
for each of one or more of the simplified query terms, identifying one or more synonyms for the simplified query term based on (i) a respective probability that each term in the search query is in a respective language based on counts of occurrences of the term in documents in each of a plurality of languages, and (ii) a frequency of the synonym in the documents in each of the plurality of languages wherein the probability that the term in the search query is in a respective language is calculated based at least in part on a ratio between the count and a number of the plurality of languages; and
augmenting the search query with one or more of the identified possible synonyms.
8. The program product of claim 7 wherein the calculation of the probability includes application of a smoothing factor to reduce noise.
9. The program product of claim 8 wherein the smoothing factor is based on counts of occurrences of each term and the number of languages.
10. The program product of claim 7 wherein augmenting the query further comprises appending the one or more identified synonyms for a simplified query term to the search query wherein the one or more identified synonyms are appended as a conjunction with the simplified query term.
11. The program product of claim 7 wherein the one or more mappings comprise one or more character conversion maps.
12. The program product of claim 11 wherein the one or more character conversion maps comprise conversion maps for mapping words that include collapsible digraphs or language specific conversion maps.
13. A system comprising:
a machine readable storage device having instructions stored thereon; and
data processing apparatus operable to execute the instructions to perform operations comprising:
receiving by way of a user interface, a search query comprising one or more query terms, the user interface having an interface language that is a natural language in which the user interface presents information;
based on the interface language, selecting one or more mappings and using the selected one or more mappings to simplify each query term to a corresponding simplified query term;
for each of one or more of the simplified query terms, identifying one or more synonyms for the simplified query term based on (i) a respective probability that each term in the search query is in a respective language based on counts of occurrences of the term in documents in each of a plurality of languages, and (ii) a frequency of the synonym in the documents in each of the plurality of languages wherein the probability that the term in the search query is in a respective language is calculated based at least in part on a ratio between the count and a number of the plurality of languages; and
augmenting the search query with one or more of the identified possible synonyms.
14. The system of claim 13 wherein the calculation of the probability includes application of a smoothing factor to reduce noise.
15. The system of claim 14 wherein the smoothing factor is based on counts of occurrences of each term and the number of languages.
16. The system of claim 15 wherein augmenting the query further comprises appending the one or more identified synonyms for a simplified query term to the search query wherein the one or more identified synonyms are appended as a conjunction with the simplified query term.
17. The system of claim 13 wherein the one or more mappings comprise one or more character conversion maps.
18. The system of claim 17 wherein the one or more character conversion maps comprise conversion maps for mapping words that include collapsible digraphs or language specific conversion maps.