System and method for locating bilingual web sites
Disclosed herein are systems, methods, and non-transitory computer-readable storage media for bootstrapping a language translation system. A system configured to practice the method performs a bidirectional web crawl to identify a bilingual website. The system analyzes data on the bilingual website to make a classification decision about whether the root of the bilingual website is an entry point for the bilingual website. The bilingual site can contain pairs of parallel pages. Each pair can include a first website in a first language and a second website in a second language, and a first portion of the first web page corresponds to a second portion of the second web page. Then the system analyzes the first and second web pages to identify corresponding information pairs in the first and second languages, and extracts the corresponding information pairs from the first and second web pages for use in a language translation model.
1. A method comprising:
performing a generic web crawl to identify a first webpage in a first language having a link thereon which points to a second webpage in a second language, wherein the first webpage and the second webpage comprise a bilingual website;
based on an analysis of parameters on the first webpage comprising at least two of: the link pointing to the second webpage, a title, a link neighborhood, a link context and data indicating a separate version of the first webpage, classifying the first webpage as a root page and as an entry point for the bilingual website via the link to the second webpage;
performing a bidirectional web crawl between the first webpage and the second webpage to identify the first webpage and the second webpage as the bilingual website, the bidirectional web crawl utilizing classifications of links to avoid links having a low respective relevance;
extracting information pairs from the first webpage and the second webpage for use in a language translation model, the information pairs comprising at least one of a word pair, a paragraph pair and a sentence pair; and
updating a statistical model with domain representative data using the information pairs.
2. The method of claim 1 , further comprising:
identifying, using a visitation policy which constrains web-crawling to a graph neighborhood of bilingual websites, a pattern of links within between the first webpage and the second webpage, to yield a bipartite graph; and
ranking a relevance of candidate links which point to parallel text in the first webpage and the second webpage, to yield the classifications, wherein the classifications comprise classifications of links based on the bipartite graph.
3. The method of claim 2 , wherein the relevance is based on supervised learning via a support vector machine and a link predictor, and wherein the link predictor filters irrelevant pages when the irrelevant pages have less than a threshold amount of relevant links.
4. The method of claim 1 , further comprising bootstrapping the language translation model using the information pairs.
5. The method of claim 1 , wherein identification of the bilingual website comprises identifying a pair of parallel pages.
6. The method of claim 1 , wherein the bidirectional web crawl considers back links and forward links.
7. The method of claim 6 , wherein each of the back links and the forward links is associated with a relevance score.
8. The method of claim 7 , wherein the relevance score is based on a context of a link in a neighborhood of elements.
9. The method of claim 1 , wherein a frontier scheduler generates a list of links for use in the bidirectional web crawl.
10. The method of claim 1 , further comprising augmenting a statistical model with domain representative data based on the information pairs.
11. The method of claim 1 , wherein the language translation model is one of a machine translation model, a cross-lingual document retrieval model, and a language model.
12. A system comprising:
a processor;
a computer-readable storage memory having instructions stored which, when executed by the processor, cause the processor to perform operations comprising:
performing a generic web crawl to identify a first webpage in a first language having a link thereon which points to a second webpage in a second language, wherein the first webpage and the second webpage comprise a bilingual website;
based on an analysis of parameters on the first webpage comprising at least two of: the link pointing to the second webpage, a title, a link neighborhood, a link context and data indicating a separate version of the first webpage, classifying the first webpage as a root page and as an entry point for the bilingual website via the link to the second webpage;
performing a bidirectional web crawl between the first webpage and the second webpage to identify the first webpage and the second webpage as the bilingual website, the bidirectional web crawl utilizing classifications of links to avoid links having a low respective relevance;
extracting information pairs from the first webpage and the second webpage for use in a language translation model, the information pairs comprising at least one of a word pair, a paragraph pair and a sentence pair; and
updating a statistical model with domain representative data using the information pairs.
13. The system of claim 12 , the computer-readable storage memory storing additional instructions stored which, when executed by the processor, cause the processor to perform operations comprising:
identifying, using a visitation policy which constrains web-crawling to a graph neighborhood of bilingual websites, a pattern of links within between the first webpage and the second webpage, to yield a bipartite graph; and
ranking a relevance of candidate links which point to parallel text in the first webpage and the second webpage, to yield the classifications, wherein the classifications comprise classifications of links based on the bipartite graph.
14. The system of claim 13 , wherein the relevance is based on supervised learning via a support vector machine and a link predictor, and wherein the link predictor filters irrelevant pages when the irrelevant pages have less than a threshold amount of relevant links.
15. The system of claim 12 , the computer-readable storage memory storing additional instructions stored which, when executed by the processor, cause the processor to perform operations comprising:
bootstrapping the language translation model using the information pairs.
16. The system of claim 12 , wherein identification of the bilingual website comprises identifying a pair of parallel pages.
17. The system of claim 12 , wherein the bidirectional web crawl considers back links and forward links.
18. The system of claim 17 , wherein each of the back links and the forward links is associated with a relevance score.
19. The system of claim 18 , wherein the relevance score is based on a context of a link in a neighborhood of elements.
20. A computer-readable storage memory having instructions stored which, when executed by a processor, cause the processor to perform operations comprising:
performing a generic web crawl to identify a first webpage in a first language having a link thereon which points to a second webpage in a second language, wherein the first webpage and the second webpage comprise a bilingual website;
based on an analysis of parameters on the first webpage comprising at least two of: the link pointing to the second webpage, a title, a link neighborhood, a link context and data indicating a separate version of the first webpage, classifying the first webpage as a root page and as an entry point for the bilingual website via the link to the second webpage;
performing a bidirectional web crawl between the first webpage and the second webpage to identify the first webpage and the second webpage as the bilingual website, the bidirectional web crawl utilizing classifications of links to avoid links having a low respective relevance;
extracting information pairs from the first webpage and the second webpage for use in a language translation model, the information pairs comprising at least one of a word pair, a paragraph pair and a sentence pair; and
updating a statistical model with domain representative data using the information pairs.