Systems and methods of web crawling
Methods and systems for dynamically training a web crawler. The web crawler maintains one or more categories each comprising a set of words. The method includes selecting at least one hyperlink in response to a query received from a user. The method further includes determining a hyperlink score for the at least one hyperlink based on a category score associated with each of one or more categories. The category score associated with each of the one or more categories is updated based at least in part on the hyperlink score. The updated category score is compared with the hyperlink score to select a category from the one or more categories. The set of words associated with the category is updated based on content of a web page pointed by the at least one hyperlink.
1. A method for training a web crawler, wherein the web crawler maintains one or more categories each comprising a set of words, the method comprising:
in response to receiving a query from a user:
selecting, by a processor, at least one hyperlink based on the set of words;
determining, by the processor, a hyperlink score for the at least one hyperlink based on a predetermined category score associated with each of one or more categories and a membership value of the at least one hyperlink for each of the one or more categories;
updating, by the processor, the predetermined category score associated with each of the one or more categories based at least on a discount factor associated with the predetermined category score and an association of learning rate with a measure of contribution of the one or more categories for the selection of the at least one hyperlink and another measure of correctness of the selection of the at least one hyperlink with respect to semantic of the query;
comparing, by the processor, the updated predetermined category score with the hyperlink score to select a category from the one or more categories; and
updating, by the processor, the set of words associated with the category based on content of a web page pointed by the at least one hyperlink.
2. The method of claim 1 further comprising determining, by the processor, a first reward score for the at least one hyperlink based on a predetermined scaling value and a relevancy score of the at least one hyperlink, wherein the relevancy score is determined based on semantics of the query.
3. The method of claim 2 further comprising determining, by the processor, a contribution score of each of the one or more categories based on the hyperlink score, the predetermined category score, and the membership value, wherein the contribution score is indicative of contribution of each of the one or more categories in selection of the at least one hyperlink.
4. The method of claim 3 further comprising determining, by the processor, a second reward score for each of the one or more categories based on the contribution score and the first reward score.
5. The method of claim 4 , wherein the predetermined category score is updated based on the second reward score.
6. The method of claim 2 further comprising updating, by the processor, the hyperlink score based on the first reward score.
7. A system for training a web crawler, wherein the web crawler maintains one or more categories each comprising a set of words, the system comprises:
one or more processors operable to execute one or more instructions to:
select at least one hyperlink in response to a query received from a user based on the set of words;
determine a hyperlink score for the at least one hyperlink based on a predetermined category score associated with each of one or more categories and a membership value of the at least one hyperlink for each of the one or more categories;
update the predetermined category score associated with each of the one or more categories based at least on a discount factor associated with the predetermined category score and an association of learning rate with a measure of contribution of the one or more categories for the selection of the at least one hyperlink and another measure of correctness of the selection of the at least one hyperlink with respect to semantic of the query;
compare the updated predetermined category score with the hyperlink score to select a category from the one or more categories; and
update the set of words associated with the category based on content of a webpage pointed by the at least one hyperlink.
8. The system of claim 7 , wherein the one or more processors is further operable to execute the one or more instructions to determine a first reward score for the at least one hyperlink based on a predetermined scaling value and a relevancy score of the at least one hyperlink, wherein the relevancy score is determined based on semantics of the query.
9. The system of claim 8 , wherein the one or more processors is further operable to execute the one or more instructions to determine a contribution score of each of the one or more categories based on the hyperlink score, the predetermined category score, and the membership value, wherein the contribution score is indicative of contribution of each of the one or more categories in selection of the at least one hyperlink.
10. The system of claim 9 , wherein the one or more processors is further operable to execute the one or more instructions to determine a second reward score for each of the one or more categories based on the contribution score and the first reward score.
11. The system of claim 10 , wherein the predetermined category score is updated based on the second reward score.
12. The system of claim 8 , wherein the one or more processors is further operable to execute the one or more instructions to update the hyperlink score based on the first reward score.
13. The system of claim 7 , wherein the content of the webpage comprises one or more of an anchor, a header, and a page title of a web page.
14. A computer program product for use with a computer, the computer program product comprising a non-transitory computer readable medium, the non-transitory computer readable medium stores a computer program code for training a web crawler, wherein the web crawler maintains one or more categories each comprising a set of words, the computer program code is executable by a processor to:
select at least one hyperlink in response to a query received from a user based on the set of words;
determine a hyperlink score for the at least one hyperlink based on a predetermined category score associated with each of one or more categories and a membership value of the at least one hyperlink for each of the one or more categories;
update the predetermined category score associated with each of the one or more categories based at least on a discount factor associated with the predetermined category score and an association of learning rate with a measure of contribution of the one or more categories for the selection of the at least one hyperlink and another measure of correctness of the selection of the at least one hyperlink with respect to semantic of the query;
compare the updated predetermined category score with the hyperlink score to select a category from the one or more categories; and
update the set of words associated with the category based on content of a webpage pointed by the at least one hyperlink.