Automatic crawling of encoded dynamic URLs
View Patent ↗A computer program product for crawling URLs that are encoded and highly dynamic, the computer program product includes a non-transitory computer readable storage medium having computer readable program code embodied therewith. The computer readable program code includes computer readable program code configured to retrieve navigational state information corresponding to a URL and compare the navigational state information to previously stored navigational state information corresponding to one or more previously visited URLs. The computer readable program code also includes computer readable program code configured to determine if the URL has been previously visited and retrieve content associated with the URL if the URL has not been previously visited.
1. A computer program product for crawling URLs that are encoded and highly dynamic, the computer program product comprising:
a non-transitory computer readable storage medium having computer readable program code embodied therewith, the computer readable program code comprising:
computer readable program code configured to:
receiving an encoded URL corresponding to a web-portal page hosted by a web-portal server, wherein a text of the encoded URL does not provide information about content of the web-portal page, the content comprising a plurality of re-arrangeable portlets, and wherein the web-portal server provides, via a representational state transfer service, to a web-crawler, a navigational state information of the web-portal page using a predetermined format;
initializing a list of parameters, wherein the list identifies that the web-portal page corresponding to the encoded URL has been visited, and determining the parameters to add to the list by:
decoding the encoded URL by sending the encoded URL to the representational state transfer service from the web-portal server hosting the web-portal page;
receiving the navigational state information of the web-portal page from the representational state transfer service, wherein the navigational state information is decoded from the encoded URL, and comprises a selection-node-ID and a resource-ID, the selection-node-ID being an identifier assigned to the web-portal page, and the resource-ID being an identifier associated with a resource used by the web-portal page, wherein,
determining a type of the encoded URL based on the received representational state information, and
when the type is determined to be a resource URL, storing the resource-ID in the list of parameters; and
when the type is determined to be an engine URL, storing the selection-node-ID in the list of parameters, and when the type is determined to be a portlet, storing information of the portlet in the list of parameters;
determining if the encoded URL has been previously visited based on the list of parameters; and
retrieving the content associated with the encoded URL if the encoded URL has not been previously visited.
2. The computer program product of claim 1 , wherein the computer readable program code is further configured to:
analyze content associated with the encoded URL; and
explore an additional URL found in the content associated with the encoded URL.
3. The computer program product of claim 2 , wherein the computer readable program code is further configured to detect if the additional URL is a logout link prior to exploring the additional URL.
4. The computer program product of claim 1 , wherein the navigational state information from the representational state transfer service is received in XML format.
5. The computer program product of claim 1 , wherein the navigational state information includes information indicative of the type of the encoded URL.
6. The system of claim 1 , wherein the web-portal server uses websphere portal framework.
7. A system, comprising:
a computing network including a processing device in communication with one or more non-transitory computer memory storage devices; and
the computing network further configured to implement a method comprising:
receiving an encoded URL corresponding to a web-portal page hosted by a web-portal server, wherein a text of the encoded URL does not provide information about content of the web-portal page, the content comprising a plurality of re-arrangeable portlets, and wherein the web-portal server provides, via a representational state transfer service, to a web-crawler, a navigational state information of the web-portal page using a predetermined format;
initializing a list of parameters, wherein the list identifies that the web-portal page corresponding to the encoded URL has been visited, and determining the parameters to add to the list by:
decoding the encoded URL by sending the encoded URL to the representational state transfer service from the web-portal server hosting the web-portal page;
receiving the navigational state information of the web-portal page from the representational state transfer service, wherein the navigational state information is decoded from the encoded URL, and comprises a selection-node-ID and a resource-ID, the selection-node-ID being an identifier assigned to the web-portal page, and the resource-ID being an identifier associated with a resource used by the web-portal page, wherein,
determining a type of the encoded URL based on the received representational state information, and
when the type is determined to be a resource URL, storing the resource-ID in the list of parameters; and
when the type is determined to be an engine URL, storing the selection-node-ID in the list of parameters, and when the type is determined to be a portlet, storing information of the portlet in the list of parameters;
determining if the encoded URL has been previously visited based on the list of parameters; and
retrieving the content associated with the encoded URL if the encoded URL has not been previously visited.
8. The system of claim 7 , wherein the web-portal server uses websphere portal framework.