System and method for building diverse language models
Disclosed herein are systems, methods, and non-transitory computer-readable storage media for collecting web data in order to create diverse language models. A system configured to practice the method first crawls, such as via a crawler operating on a computing device, a set of documents in a network of interconnected devices according to a visitation policy, wherein the visitation policy is configured to focus on novelty regions for a current language model built from previous crawling cycles by crawling documents whose vocabulary considered likely to fill gaps in the current language model. A language model from a previous cycle can be used to guide the creation of a language model in the following cycle. The novelty regions can include documents with high perplexity values over the current language model.
1. A method comprising:
establishing a web-page analysis module configured to identify a likelihood of web pages having information relevant for a language model, and wherein the web-page analysis module is further configured to determine whether or not to crawl a respective web page based at least in part on a perplexity value associated with a configuration of the respective web page relative to a vocabulary on the respective web page to yield a crawling schedule;
crawling, via a processor, the web pages based on the crawling schedule, to yield new vocabulary words; and
generating a new language model based at least in part on the new vocabulary words.
2. The method of claim 1 , wherein the new language model is not based on a previous language model.
3. The method of claim 1 , wherein the new language model comprises an update of a previous language model.
4. The method of claim 1 , wherein the information relevant for the language model relates to a vocabulary gap in the language model.
5. The method of claim 1 , further comprising recognizing received speech using the new language model.
6. The method of claim 1 , wherein the new language model is generated by modifying the language model.
7. The method of claim 1 , further comprising updating a website visitation policy for the crawling once a specified number of pages is crawled.
8. The method of claim 1 , wherein the new language model is generated from a merging of a set of language models.
9. A system comprising:
a processor; and
a computer-readable storage medium having instructions stored which, when executed by the processor, cause the processor to perform instructions comprising:
establishing a web-page analysis module configured to identify a likelihood of web pages having information relevant for a language model, and wherein the web-page analysis module is further configured to determine whether or not to crawl a respective web page based at least in part on a perplexity value associated with a configuration of the respective web page relative to a vocabulary on the respective web page to yield a crawling schedule;
crawling the web pages based on the crawling schedule, to yield new vocabulary words; and
generating a new language model based at least in part on the new vocabulary words.
10. The system of claim 9 , wherein the new language model is not based on a previous language model.
11. The system of claim 9 , wherein the new language model comprises an update of a previous language model.
12. The system of claim 9 , wherein the information relevant for the language model relates to a vocabulary gap in the language model.
13. The system of claim 9 , wherein the computer-readable storage medium stores additional instructions stored which, when executed by the processor, cause the processor to perform instructions further comprising:
recognizing received speech using the new language model.
14. The system of claim 9 , wherein the new language model is generated by modifying the language model.
15. The system of claim 9 , wherein the computer-readable storage medium stores additional instructions stored which, when executed by the processor, cause the processor to perform instructions further comprising:
updating a website visitation policy for the crawling once a specified number of pages is crawled.
16. The system of claim 9 , wherein the new language model is generated from a merging of a set of language models.
17. A non-transitory computer-readable storage device having instructions stored which, when executed by a computing device, cause the computing device to perform operations comprising:
establishing a web-page analysis module configured to identify a likelihood of web pages having information relevant for a language model, and wherein the web-page analysis module is further configured to determine whether or not to crawl a respective web page based at least in part on a perplexity value associated with a configuration of the respective web page relative to a vocabulary on the respective web page to yield a crawling schedule;
crawling the web pages based on the crawling schedule, to yield new vocabulary words; and
generating a new language model based at least in part on the new vocabulary words.
18. The non-transitory computer-readable storage device of claim 17 , wherein the new language model is not based on a previous language model.
19. The non-transitory computer-readable storage device of claim 17 , wherein the new language model comprises an update of a previous language model.
20. The non-transitory computer-readable storage device of claim 17 , wherein the information relevant for the language model relates to a vocabulary gap in the language model.