System and method for hybrid multilingual search indexing
System and method for the indexing and searching of multilingual documents are disclosed.
1. A system, comprising:
a processor; and
a computer readable medium storing instructions translatable by the processor to implement a multilingual search engine, comprising instructions for:
receiving a multilingual document;
determining a set of fragments of a content of the multilingual document, wherein each of the set of fragments comprises a portion of the content of the multilingual document;
determining an index language associated with each fragment of the determined set of fragments of the multilingual document;
based on the index language, indexing each fragment in a multilingual object index in association with the multilingual document, wherein the multilingual object index includes tokens from the multilingual document in multiple languages determined according to their respective index language;
receiving a search query; and
performing the search query by:
determining a set of search fragments of the search query, wherein each of the set of search fragments comprises a portion of the search query;
determining a search language associated with each search fragment of the determined set of search fragments; and
based on the search language associated with each search fragment, indexing each search fragment in a multilingual search index in association with the search query; and
performing a search of the multilingual object index according to the search query based on the multilingual search index.
2. The system of claim 1 , wherein indexing each fragment is performed using a language specific model for the index language.
3. The system of claim 2 , wherein the indexing of each search fragment is preformed using a language specific model for the search language.
4. The system of claim 3 , wherein the language specific model for the index language is different that then language specific model for the search language.
5. The system of claim 1 , wherein each fragment is indexed in a single field.
6. The system of claim 1 , wherein the multilingual search index comprises tokens of the search fragments according to their respective associated search language.
7. The system of claim 6 , wherein the multilingual search index is temporarily stored while performing the search.
8. A method, comprising:
receiving a multilingual document;
determining a set of fragments of a content of the multilingual document, wherein each of the set of fragments comprises a portion of the content of the multilingual document;
determining an index language associated with each fragment of the determined set of fragments of the multilingual document;
based on the index language, indexing each fragment in a multilingual object index in association with the multilingual document, wherein the multilingual object index includes tokens from the multilingual document in multiple languages determined according to their respective index language;
receiving a search query; and
performing the search query by:
determining a set of search fragments of the search query, wherein each of the set of search fragments comprises a portion of the search query;
determining a search language associated with each search fragment of the determined set of search fragments; and
based on the search language associated with each search fragment, indexing each search fragment in a multilingual search index in association with the search query; and
performing a search of the multilingual object index according to the search query based on the multilingual search index.
9. The method of claim 8 , wherein indexing each fragment is performed using a language specific model for the index language.
10. The method of claim 9 , wherein the indexing of each search fragment is preformed using a language specific model for the search language.
11. The method of claim 10 , wherein the language specific model for the index language is different than language specific model for the search language.
12. The method of claim 8 , wherein each fragment is indexed in a single field.
13. The method of claim 8 , wherein the multilingual search index comprises tokens of the search fragments according to their respective associated search language.
14. The method of claim 13 , wherein the multilingual search index is temporarily stored while performing the search.
15. A non-transitory computer readable medium, comprising instructions for:
receiving a multilingual document;
determining a set of fragments of a content of the multilingual document, wherein each of the set of fragments comprises a portion of the content of the multilingual document;
determining an index language associated with each fragment of the determined set of fragments of the multilingual document;
based on the index language, indexing each fragment in a multilingual object index in association with the multilingual document, wherein the multilingual object index includes tokens from the multilingual document in multiple languages determined according to their respective index language;
receiving a search query; and
performing the search query by:
determining a set of search fragments of the search query, wherein each of the set of search fragments comprises a portion of the search query;
determining a search language associated with each search fragment of the determined set of search fragments; and
based on the search language associated with each search fragment, indexing each search fragment in a multilingual search index in association with the search query; and
performing a search of the multilingual object index according to the search query based on the multilingual search index.
16. The non-transitory computer readable medium of claim 15 , wherein indexing each fragment is performed using a language specific model for the index language.
17. The non-transitory computer readable medium of claim 16 , wherein the indexing of each search fragment is preformed using a language specific model for the search language.
18. The non-transitory computer readable medium of claim 17 , wherein the language specific model for the index language is different than then language specific model for the search language.
19. The non-transitory computer readable medium of claim 15 , wherein each fragment is indexed in a single field.
20. The non-transitory computer readable medium of claim 15 , wherein the multilingual search index comprises tokens of the search fragments according to their respective associated search language.
21. The non-transitory computer readable medium of claim 20 , wherein the multilingual search index is temporarily stored while performing the search.