System and method for identifying cloaked web servers
View Patent ↗A search engine receives from a client a representation of a first object that was returned by a web server to the client in response to a request from the client. The search engine receives from the web server a second object in response to an identical request from the search engine, and compares the representation of the first object to a representation of the second object. The web server is determined to be cloaked if the representation of the first object does not match the representation of the second object. Typically, the client receives a URL embedded in a response to a search request submitted to the search engine. A toolbar operating in conjunction with the web browser on the client processes the URL. The processing includes: directing the web browser to obtain an object corresponding to the URL from a web server addressed by the URL; converting the object to a feature vector; and delivering the feature vector and the URL back to the search engine.
1. A method for identifying a cloaked web server comprising:
receiving from a client computer a representation of a first object, said first object returned by a web server to said client computer in response to a first request from said client computer for an object corresponding to a network address, said web server addressed by said network address;
receiving from the web server a second object in response to a second request for the object corresponding to said network address; and
comparing the representation of the first object to a representation of the second object, and determining that the web server is cloaked if said representation of the first object does not match said representation of the second object.
2. The method of claim 1 , further comprising
sending to the client computer the network address prior to receiving from the client computer the representation of the first object.
3. The method of claim 2 , further comprising
selecting the network address from a set of network addresses according to a predefined function.
4. The method of claim 2 , further comprising
periodically sending one of a set of network addresses to the client computer according to a predefined schedule, the network address being selected from said set of network addresses.
5. The method of claim 2 , further comprising
receiving from the client computer a request for the network address prior to sending the network address to the client computer.
6. The method of claim 2 , further comprising
receiving from the client computer a request for search engine services prior to sending the network address to the client computer; and
sending to the client computer a response to the request for the search engine services, said response to the request for the search engine services including the network address.
7. The method of claim 6 , wherein
the request for search engine services includes one or more search terms; and
the response to the request for search engine services includes one or more network addresses corresponding to the one or more search terms.
8. The method of claim 7 , wherein
the network address corresponds to the one or more search terms.
9. The method of claim 7 , further comprising
when the web server is determined to be cloaked, excluding from the response to the request network addresses addressing the web server.
10. The method of claim 6 , wherein
the response to the request for search engine services includes an instruction directing the client computer to return the representation of the first object.
11. The method of claim 1 , wherein
the representation of the first object is a first feature vector comprising a first set of features;
the representation of the second object is a second feature vector comprising a second set of features; and
the comparing and determining including determining whether the second feature vector has at least a predetermined number of features in common with the first feature vector.
12. The method of claim 1 , further comprising
maintaining a list of network addresses corresponding to a search term, said list of network addresses including the network address; and
removing from the list of network addresses the network address if the representation of the first object does not match the representation of the second object.
13. The method of claim 1 , further comprising
maintaining a list of cloaked web servers; and
adding to the list of cloaked web servers the web server if the representation of the first object does not match the representation of the second object.
14. A computer system for identifying a cloaked web server comprising:
a central processing unit, an instruction module, and a network interface;
the instruction module being executable by the central processing unit and including
client interface instructions for receiving from a client computer through the network interface a representation of a first object, said first object returned by a web server to said client computer in response to a first request from said client computer for an object corresponding to a network address, said web server addressed by said network address;
web server interface instructions for receiving through the network interface a second object returned by the web server in response to a second request from the computer system for the object corresponding to the network address; and
compare instructions for comparing the representation of the first object to a representation of the second object and for determining that the web server is cloaked if said representation of the first object does not match said representation of the second object.
15. The computer system of claim 14 , the instruction module further comprising
additional client interface instructions for sending the network address to the client computer prior to receiving the representation of the first object from said client computer.
16. The computer system of claim 15 , the instruction module further comprising
instructions for selecting the network address from a set of network addresses according to a predefined function.
17. The computer system of claim 16 , the instruction module further comprising
instructions for periodically sending one of a set of network addresses to the client computer according to a predefined schedule, the network address being selected from said set of network addresses.
18. The computer system of claim 15 , the instruction module further comprising
instructions for receiving from the client computer a request for the network address prior to sending the network address to the client computer.
19. The computer system of claim 15 , the instruction module further comprising
instructions for receiving from the client computer a request for search engine services prior to sending the network address to the client computer; and
instructions for sending to the client computer a response to the request for the search engine services, said response to the request for the search engine services including the network address.
20. The computer system of claim 19 , wherein
the computer system comprises a web server configured to provide search engine services to one or more client computers.
21. The computer system of claim 19 , wherein
the request for search engine services includes one or more search terms; and
the response to the request for search engine services includes one or more network addresses corresponding to the one or more search terms.
22. The computer system of claim 21 , wherein
the network address corresponds to the one or more search terms.
23. The computer system of claim 19 , wherein
the response to the request for search engine services includes an instruction directing the client computer to return the representation of the first object.
24. The computer system of claim 19 , the instruction module further comprising
instructions for deriving the representation of the second object;
wherein
the representation of the first object is a first feature vector comprising a first set of features;
the representation of the second object is a second feature vector comprising a second set of features; and
the compare instructions include instructions for determining whether the second feature vector has at least a predetermined number of features in common with the first feature vector.
25. The computer system of claim 19 , the instruction module further comprising
instructions for maintaining a list of network addresses corresponding to a search term, said list of network addresses including the network address; and
instructions for removing from the list of network addresses the network address if the representation of the first object does not match the representation of the second object.
26. The computer system of claim 14 , the instruction module further comprising
instructions for maintaining a list of network addresses corresponding to a search term; and
instructions for removing from the list of network addresses each network address corresponding to the web server if said web server is cloaked.
27. The computer system of claim 14 , the instruction module further comprising
instructions for maintaining a list of cloaked web servers; and
instructions for adding to the list of cloaked web servers the web server if said web server is determined to be cloaked.
28. A computer program product for use in conjunction with a computer system, the computer program product comprising a computer readable storage medium and a computer program mechanism embedded therein, the computer program mechanism comprising:
client interface instructions for receiving from a client computer a representation of a first object, said first object returned by a web server to said client computer in response to a first request from said client computer for an object corresponding to a network address, said web server addressed by said network address;
web server interface instructions for receiving a second object returned by the web server in response to a second request from said computer system for the object corresponding to said network address; and
compare instructions for comparing the representation of the first object to a representation of the second object to determine if the representation of the first object matches the representation of the second object; and
cloak determination instructions for determining that the web server is cloaked if said representation of the first object does not match said representation of the second object.