System and method for building and retrieving a full text index
View Patent ↗An indexing engine generates a full text index of English and non-English files provided to the indexing engine. The indexing engine receives an input file for indexing, and normalizes the unique words contained in the input file. The normalizing includes stripping the words of any diacritical marks, taking into account different multilingual issues, case folding the words into lowercase, and the like. The normalized words are stored in a dictionary, and a word record is generated for each stored word. Each word record includes a flag that indicates whether one or more variations exist in the input file for the normalized word. One or more tables store information on the variations for the normalized words. When a query engine is invoked to search for an input query word, the variations are searched only if the user has set an option to consider such variations.
1. An indexing method comprising:
receiving an input file for indexing, the input file including a word including a diacritic symbol, the diacritic symbol being a mark added to a character;
storing information on the diacritic symbol in a first data structure, wherein the information includes a numeric value indicative of a position of the diacritic symbol in the word, and a representation of the diacritic symbol;
stripping the word of the diacritic symbol; and
storing the stripped word in a second data structure,
wherein, a query engine is configured to receive an input query word and search the second data structure for an indexed word matching the input query word, wherein in response to a command to consider diacritic symbols, the query engine is further configured to retrieve the information on the diacritic symbol corresponding to the matched indexed word from the first data structure, and compare the retrieved information against information of a diacritic symbol in the input query word for determining a match.
2. The method of claim 1 , wherein the input query word includes the diacritic symbol, and the query engine is further configured to:
strip the input query word of the diacritic symbol; and
search the second data structure for the stripped input query word.
3. The method of claim 2 , wherein the query engine is further configured to:
search the first data structure for a match of the diacritic symbol stripped from the input query word if the input query word is associated with a command to consider diacritics.
4. The method of claim 2 , wherein the query engine is configured to receive a command to consider diacritics in the input query word, the method further comprising:
storing second information on the diacritic symbol stripped from the input query word, wherein the second information includes a second numeric value indicative of a position of the diacritic symbol in the input query word, and a representation of the diacritic symbol,
wherein, in response to the command, the query engine is configured to search the first data structure for information on the diacritic symbol for an indexed word corresponding to the stripped input query word, compare the information on the diacritic symbol with the second information on the diacritic symbol stripped from the input query word, and return a no match in response to a no match of the compared information.
5. The method of claim 1 , wherein the representation is a mark representing the diacritic symbol.
6. The method of claim 1 , wherein the representation is an index to a table storing a mark representing the diacritic symbol.
7. The method of claim 1 , wherein the first data structure stores the information on the diacritic symbol separately from the word.
8. An indexing method comprising:
receiving an input file for indexing, the input file including a plurality of words, at least one of the plurality of words including a diacritic symbol, the diacritic symbol being a mark added to a character;
storing information on the diacritic symbol in a first data structure, wherein the information includes a numeric value indicative of a position of the diacritic symbol in the word, and a representation of the diacritic symbol;
normalizing the plurality of words contained in the input file;
storing the plurality of normalized words in a word table;
generating a word record for each of the plurality of normalized words in the word table, each word record including one or more flags for indicating whether one or more variations exist in the file for the normalized word,
wherein, a query engine is configured to receive an input query word, normalize the input query word, and search the word table for a match of the normalized input query word with a stored normalized word, wherein in response to a command to consider diacritic symbols, the query engine is further configured to retrieve the information on the diacritic symbol corresponding to the matched normalized word from the first data structure, and compare the retrieved information against information of a diacritic symbol in the input query word for determining a match.
9. The method of claim 8 , wherein normalizing one of the plurality of words includes stripping the word of a diacritic symbol.
10. The method of claim 8 , wherein normalizing one of the plurality of words includes replacing a particular character of the word with a predefined character.
11. The method of claim 8 further comprising:
generating one or more tables storing information on the one or more variations for each of the normalized words, wherein, the query engine is configured to search the one or more tables for information on the one or more variations matching the input query word responsive to a command to consider the one or more variations.
12. The method of claim 11 , wherein the variation of one of the normalized words includes a diacritic symbol, and the table stores information on the diacritic symbol.
13. The method of claim 11 , wherein the variation of one of the normalized words includes a casing variation.
14. A method for processing a search query for a query word, the method comprising:
receiving an input query word;
normalizing the input query word;
searching a word table for the normalized input query word, the word table storing a plurality of normalized words associated with a file;
retrieving from the word table information on a word record for a normalized word matching the normalized input query word, the word record including a flag for indicating whether one or more variations exist in the file for the normalized word, wherein in response to a command to consider diacritic symbols, information on a diacritic symbol corresponding to the matched normalized word is retrieved from a first data store and compared against information of a diacritic symbol in the input query word for determining a match, wherein the information on the diacritic symbol stored in the first data structure includes a numeric value indicative of a position of the diacritic symbol in the word prior to normalizing, and a representation of the diacritic symbol, the diacritic symbol being a mark added to a character.
15. The method of claim 14 , wherein normalizing the query word includes stripping the query word of a diacritic symbol.
16. The method of claim 14 , wherein normalizing the query word includes replacing a particular character with a predefined character.
17. The method of claim 14 , wherein one or more tables store information on the one or more variations of the normalized word, the method further comprising:
searching the one or more tables for information on the one or more variations matching the input query word responsive to the command to consider the one or more variations.
18. The method of claim 17 , wherein one the variations of the normalized word includes a diacritic symbol, and the table stores information on the diacritic symbol.
19. The method of claim 17 , wherein one of the variations of the normalized word is a casing variation.
20. An indexing system comprising:
an input device configured to receive an input file for indexing;
a processor configured to normalize a plurality of words contained in the input file, at least one of the plurality of words including a diacritic symbol, the diacritic symbol being a mark added to a character;
at least one data storage device storing:
a word table storing the plurality of normalized words;
a diacritical table storing information on the diacritic symbol included in the at least one of the plurality of words, the information including a numeric value indicative of a position of the diacritic symbol in the word, and a representation of the diacritic symbol; and
a word record for each of the plurality of normalized words, each word record including a flag for indicating whether one or more variations exist in the input file for the normalized word,
wherein, a query engine is configured to receive an input query word, normalize the input query word, and search the word table for a match of the normalized input query word with a stored normalized word, wherein in response to a command to consider diacritic symbols, the query engine is further configured to retrieve the information on the diacritic symbol corresponding to the matched normalized word from the first data structure, and compare the retrieved information against information of a diacritic symbol in the input query word for determining a match.
21. The system of claim 20 , wherein the means for normalizing one of the plurality of words includes stripping the word of a diacritic symbol.
22. The system of claim 20 , wherein the means for normalizing one of the plurality of words includes replacing a particular character of the word with a predefined character.
23. The system of claim 20 , wherein the data storage device further stores one or more tables storing information on the one or more variations for each of the normalized words, wherein, the query engine is configured to search the one or more tables for information on the one or more variations matching the input query word responsive to a command to consider the one or more variations.
24. The system of claim 20 , wherein the flag indicates the variation as being inclusion of a diacritic symbol.
25. The system of claim 20 , wherein the one or more variations for the normalized word includes a casing variation.
26. A system for processing a search query for a query word, the system comprising:
at least one storage device storing a word table storing a plurality of normalized words associated with a file, and a word record for each normalized word stored in the word table, each word record including a flag for indicating whether one or more variations exist in the file for the normalized word;
a processor; and
a memory operably coupled to the processor and storing program instructions therein, the processor being operable to execute the program instructions, the program instructions including:
receiving an input query word;
normalizing the input query word;
searching the word table for the normalized input query word; and
retrieving from the word table information on the word record for the normalized word matching the normalized input query word, wherein the one or more variations indicated in the retrieved word record are considered for a match against the query word responsive to a command to consider the one or more variations, wherein in response to a command to consider diacritic symbols, information on a diacritic symbol corresponding to the matched normalized word is retrieved from a first data store and compared against information of a diacritic symbol in the input query word for determining a match, wherein the information on the diacritic symbol stored in the first data structure includes a numeric value indicative of a position of the diacritic symbol in the word prior to normalizing, and a representation of the diacritic symbol, the diacritic symbol being a mark added to a character.
27. The system of claim 26 , wherein the normalizing the query word includes stripping the word of a diacritic symbol.
28. The system of claim 26 further comprising:
one or more tables storing information on the one or more variations indicated in the retrieved word record, wherein the program instructions further include searching the one or more tables for information on the one or more variations matching the input query word responsive to the command to consider the one or more variations.
29. The system of claim 28 , wherein one of the variations indicated in the retrieved word record is inclusion of a diacritic symbol, and the table stores information on the diacritic symbol.