System, method, and computer program product for crawling a website based on a scheme of the website
View Patent ↗A system, method, and computer program product are provided for crawling a website based on a scheme of the website. In use, a difference between a first content and second content of a website is identified. Additionally, a scheme of the website is identified based on the difference. Furthermore, the website is crawled based on the scheme.
1. A non-transitory computer readable medium comprising instructions stored thereon that when executed by one or more processors cause the one or more processors to:
determine a difference between a first content and a second content of a single website, the first content and second content identified in a first and second probe of the single website, each probe comprising a request and a corresponding response for information;
identify a scheme of the single website based on the difference; and
direct crawling of a portion of the single website, the directed crawling based on the identified scheme, and the directed crawling to further identify vulnerability to unwanted activity on the single website.
2. The non-transitory computer readable medium of claim 1 , wherein the first content includes a first response to a first request and the second content includes a second response to a second request.
3. The non-transitory computer readable medium of claim 2 , further comprising instructions to cause the one or more processors to compare the first response and the second response to determine a severity of difference between the first response and the second response.
4. The non-transitory computer readable medium of claim 1 , wherein the difference is represented by a score.
5. The non-transitory computer readable medium of claim 1 , wherein the difference is determined utilizing a b-tree.
6. The non-transitory computer readable medium of claim 2 , wherein the scheme is identified by instructions to cause the one or more processors to process the first request, the second request, and a third request.
7. The non-transitory computer readable medium of claim 6 , wherein the instructions to cause the one or more processors to process the first and second requests take into account the difference between the first content and the second content.
8. The non-transitory computer readable medium of claim 6 , wherein the instructions to cause the one or more processors to process the first and second requests utilize a b-tree.
9. The non-transitory computer readable medium of claim 6 , wherein the instructions to cause the one or more processors to process the first request, the second request, and the third request determine a first score.
10. The non-transitory computer readable medium of claim 9 , wherein the scheme is further identified by instructions to cause the one or more processors to process the first request, the second request, and a fourth request.
11. The non-transitory computer readable medium of claim 10 , wherein the instructions to cause the one or more processors to process the first request, the second request, and the fourth request determine a second score.
12. The non-transitory computer readable medium of claim 11 , further comprising instructions to cause the one or more processors to compare the first score with the second score.
13. The non-transitory computer readable medium of claim 12 , further comprising instructions to cause the one or more processors to select only one of the third request and the fourth request to transmit to perform the directed crawling of the website, based on the comparison.
14. The non-transitory computer readable medium of claim 2 , wherein at least one of the first request, the second request, the first response, and the second response are compressed.
15. The non-transitory computer readable medium of claim 2 , wherein the first request, the second request, the first response, and the second response comprise a full set of requests and responses, and wherein only a subset of the full set of requests and responses are stored.
16. The non-transitory computer readable medium of claim 2 , wherein the first request, the second request, the first response, and the second response are processed utilizing multiple parallel threads.
17. The non-transitory computer readable medium of claim 1 , wherein the instructions to cause the one or more processors to direct crawling of the portion of the website include instructions to cause the one or more processors to perform the directed crawling for security purposes.
18. A method, comprising:
determining, utilizing a hardware processor, a difference between a first content and a second content of a single website, the difference determined in part by first removing at least one common sequence from said first content and said second content, the first content and second content identified in a first and second probe of the single website, each probe comprising a request and a corresponding response for information;
identifying a scheme of the single website based on the difference; and
directing crawling of a portion of the single website, the crawling directed based on the scheme, and the directed crawling to identify vulnerability to unwanted activity on the single website.
19. A non-transitory computer readable medium comprising instructions stored thereon that when executed cause one or more processors to:
remove at least one common sequence from a first content and a second content of a single website, the first content and second content retrieved in response to a first and second probe of the single website, each probe comprising a request and a corresponding response for information;
determine a difference between said first and second content after said at least one common sequence has been removed from said first and second content;
identify a scheme of the single website based on the difference; and
direct crawling of a portion of the single website, the directed crawling based on the scheme, and the directed crawling to identify vulnerability to unwanted activity on the single website.
20. The non-transitory computer readable medium of claim 19 , further comprising instructions to cause the one or more processors to:
initiate a first HTTP response to a first request associated with said first content and a second HTTP response to a second request associated with said second content; and
compare the first HTTP response and the second HTTP response to determine a severity of difference between the first HTTP response and the second HTTP response and to derive a score based on the severity of difference.