Support for international search terms—translate as you crawl
View Patent ↗A search engine server delivers search results to a web browser of a client device communicatively coupled to the search engine server via the Internet. The system identifies new web pages in a source language during crawling, translates them into a plurality of destination languages, creates reverse indexes in respective languages, and stores both reverse indexes and cache web pages in a database. Upon the entry of search strings by a user using a web browser, the search engine server responds by delivering links of web pages in the user-desired language (the language of the search string or a language chosen by the user) as well as web pages translated from a plurality of destination languages, ranked based upon popularity or other means. The search engine server contains a plurality of translators that translate new web pages, links that are obtained during crawling, in to a plurality of destination languages.
1. A search engine system that delivers search results via an Internet, the search engine system comprising:
at least one processor configured to implement:
a web crawler module, the web crawler module configured to:
crawl the Internet to gather a first web page having first text content in a first language;
identify the first language during crawling operations, at least in part, by processing a domain name;
a language processing service module, the language processing service module configured to, during the crawling operations, translate the first text content into both a second language in a form of second text content and a third language in a form of third text content using both a thesaurus database and a conjugate terms database;
at least one database structure, the database structure, during the crawling operations, storing indexed representations of each of the first text content, the second text content, and the third text content; and
a search processing service configured, in response to receiving search input in the second language, to identify within the at least one database structure at least a portion of the indexed representation of the second text content.
2. The search engine system of claim 1 , wherein the search processing service module delivers via web page data search results based at least in part on the indexed representation of the second text content, the search results being at least in part in the second language.
3. The search engine system of claim 1 , wherein the search processing service module, in response to receiving search input in the third language, identifies within the at least one database structure at least a portion of the indexed representation of the third text content.
4. The search engine system of claim 3 , wherein the search processing service module delivers via web page data search results based at least in part on the indexed representation of the third text content, the search results being at least in part in the third language.
5. The search engine system of claim 1 wherein the at least one database structure comprises a plurality of international language reverse index databases that store reverse indexes in a plurality of respective source languages.
6. The search engine system of claim 1 wherein the at least one database structure comprises a plurality of international language web page cache databases that store cache web pages in a plurality of respective source languages.
7. The search engine system of claim 1 wherein the at least one database structure comprises a plurality of translated language reverse index databases that store reverse indexes in a plurality of destination languages.
8. The search engine system of claim 1 wherein the at least one database structure comprises a plurality of translated language web page cache databases that store cache web pages in a plurality of destination languages.
9. The search engine system of claim 1 wherein the web crawler module identifies the first language, at least in part, using a conjugate international language terms database that comprises terms in the first language and their conjugates in at least the second language.
10. The search engine system of claim 1 wherein a web crawler support module identifies the first language by performing pattern recognition on the first text content.
11. A method performed by a search engine system that delivers search results via an Internet, the method comprising:
gathering by a web crawler via the Internet a first web page having first text content in a first language;
identifying the first language during crawling operations, at least in part, using a conjugate international language terms database that comprises terms in the first language and their conjugates in at least the second language;
translating the first text content into both a second language in a form of second text content and a third language in a form of third text content during the crawling operations using both a thesaurus database and a conjugate terms database;
storing, during the crawling operations, in at least one database structure indexed representations of each of the first text content, the second text content, and the third text content; and
in response to receiving search input in the second language, identifying within the at least one database structure at least a portion of the indexed representation of the second text content.
12. The method of claim 11 , further comprising delivering via web page data search results based at least in part on the indexed representation of the second text content, the search results being at least in part in the second language.
13. The method of claim 11 , further comprising, in response to receiving search input in the third language, identifying within the at least one database structure at least a portion of the indexed representation of the third text content.
14. The method of claim 13 , further comprising delivering via web page data search results based at least in part on the indexed representation of the third text content, the search results being at least in part in the third language.
15. The method of claim 11 , further comprising storing reverse indexes in a plurality of respective source languages in the at least one database structure.
16. The method of claim 11 , further comprising storing cache web pages in a plurality of respective source languages in the at least one database structure.
17. The method of claim 11 , further comprising storing reverse indexes in a plurality of destination languages in the at least one database structure.
18. The method of claim 11 , further comprising storing cache web pages in a plurality of destination languages in the at least one database structure.
19. The method of claim 11 , further comprising identifying the first language, at least in part, by processing a domain name.
20. The method of claim 11 , further comprising identifying the first language by performing pattern recognition on the first text content.