Methods and apparatus for dynamically generating web pages
View Patent ↗One embodiment of the invention is directed to automatically populating the cache of a web server. A webcrawler may send a request for a web page to the web server and receive the web page in response. The webcrawler may locate a link to a second web page in the received web page and determine if the link includes a predetermined pattern of characters. If the link includes the predetermined pattern of characters, the webcrawler may send a request to the webserver for the second web page. In another embodiment, the webcrawler may be made aware of changes to the content of a web site and revisit the web site after changes are made to repopulate the cache with the updated content.
1. A method of accessing a plurality of web pages served by a web server, the method comprising:
sending a first request to the web server for a first web page of the plurality of web pages;
receiving, from the web server, the requested first web page in response to the first request;
locating, in the first web page, a link to a second web page of the plurality of web pages;
determining whether the link includes a pattern of characters that matches a predetermined pattern of characters;
when it is determined that the link includes the pattern of characters, sending a plurality of second request to the web server for the second web page, wherein each of the plurality of second requests includes first information that emulates at least one user characteristic, wherein each of the plurality of second requests provides a different value for the first information that emulates the at least one user characteristic, and wherein each of the plurality of second requests, when processed by the web server, causes the web server to dynamically generate a different version of the second web page that includes content based, at least in part, on the value of the first information that emulates the at least one user characteristic, the web server dynamically generating each different version of the second web page using second information retrieved from a database;
after sending the plurality of second requests to the web server for the second web page, monitoring the database to determine whether the second information stored in the database has changed;
when it is determined that the second information stored in the database has changed, determining whether a change to the second information would result in a change to content included in the second web page; and
when it is determined that the change to the second information would result in a change to content included in the second web page, sending a third request to the web server for the second web page.
2. The method of claim 1 , wherein sending the request to the web server for the first web page causes the web server to cache the first web page and sending the plurality of second requests to the web server causes the web server to cache each different generated version of the second web page.
3. The method of claim 1 , wherein the at least one user characteristic includes a zipcode.
4. The method of claim 1 , wherein the act of sending the third request causes the web server to dynamically generate a version of the second web page that includes the changed second information.
5. At least one non-transitory computer readable medium, encoded with instructions that, when executed on a computer system including a web server, perform a method of accessing a plurality of web pages served by the web server, the method comprising:
sending a first request to the web server for a first web page of the plurality of web pages;
receiving, from the web server, the requested first web page in response to the first request;
locating, in the first web page, a link to a second web page of the plurality of web pages;
determining whether the link includes a pattern of characters that matches a predetermined pattern of characters; and
when it is determined that the link includes the pattern of characters, sending a plurality of second requests to the web server for the second web page, wherein each of the plurality of second requests includes first information that emulates at least one user characteristic, wherein each of the plurality of second requests provides a different value for the first information that emulates the at least one user characteristic, and wherein each of the plurality of second requests, when processed by the web server, causes the web server to dynamically generate a different version of the second web page that includes content based, at least in part, on the value of the first information that emulates the at least one user characteristic, the web server dynamically generating each different version of the second web page using second information retrieved from a database;
after sending the plurality of second requests to the web server for the second web page, monitoring the database to determine whether the second information stored in the database has changed;
when it is determined that the second information stored in the database has changed, determined whether a change to the second information would result in a change to content included in the second web page; and
when it is determined that a change to the second information would result in a change to content included in the second web page, sending a third request to the web server for the second web page.
6. The at least one non-transitory computer readable medium of claim 5 , wherein sending the request to the web server for the first web page causes the web server to cache the first web page and sending the plurality of second requests to the web server causes the web server to cache each different generated version of the second web page.
7. The at least one non-transitory computer readable medium of claim 5 , wherein the at least one user characteristic includes a zipcode.
8. The at least one non-transitory computer readable medium of claim 5 , wherein the act of sending the third request causes the web server to dynamically generate a version of the second web page including the change of content that includes the changed second information.
9. A computer system comprising:
a web server operable to provide access to a plurality of web pages; and
at least one controller operable to:
send a first request to the web server for a first web page of the plurality of web pages;
receive, from the web server, the requested first web page in response to the first request;
locate, in the first web page, a link to a second web page of the plurality of web pages;
determine whether the link includes a pattern of characters that matches a predetermined pattern of characters; and
when it is determined that the link includes the pattern of characters, send a plurality of second requests to the web server for the second web page, wherein each of the plurality of second requests includes first information that emulates at least one user characteristic, wherein each of the plurality of second requests provides a different value for the first information that emulates the at least one user characteristic, and wherein each of the plurality of second requests, when processed by the web server, causes the web server to dynamically generate a different version of the second web page that includes content based, at least in part, on the value of the first information that emulates the at least one user characteristic, the web server dynamically generating each different version of the second web page using second information retrieved from a database;
after sending the plurality of second requests to the web server for the second web page, monitor the database to determine whether the second information stored in the database had changed;
when it is determined that the second information stored in the database has changed, determine whether a change to the second information would result in a change to content included in the second web page; and
when it is determined that a change to the second information would result in a change to content included in the second web page, send a third request to the web server for the second web page.
10. The computer system of claim 9 , wherein sending the request to the web server for the first web page causes the web server to cache the first web page and sending the plurality of second requests to the web server causes the web server to cache each different generated version of the second web page.
11. The computer system of claim 9 , wherein the at least one user characteristic includes a zipcode.
12. The method of claim 1 , wherein the act of sending the first request to the web server further comprises automatically initiating sending the first request to the web server, without the first request being initiated by a human user.
13. The method of claim 1 , wherein the act of determining whether the link includes a pattern of characters that matches a predetermined pattern of characters further comprises determining, based on a set of configurable rules, whether the link includes the pattern of characters that matches the predetermined characters.
14. The method of claim 1 , wherein the information from the database comprises product pricing information.