Web crawler systems and methods to efficiently detect malicious sites
Systems and methods include receiving a list of web sites; anonymously browsing to each web site in the list; receiving a response based on the browsing; and analyzing the response to classify each web site as malicious or not based on a plurality of techniques including JavaScript (JS) obfuscation detection based on de-obfuscation. The systems and methods can further include providing a blacklist of web sites classified as malicious. The systems and methods can further include determining the list of web sites periodically based on a plurality of factors. The JS obfuscation detection can be performed by de-obfuscating JS content and utilizing heuristics to determine if the de-obfuscated JS content is malicious, and the heuristics can include a presence of any of a new JS function and a domain in the de-obfuscated JS content.
1. A non-transitory computer-readable storage medium having computer-readable code stored thereon for programming a server to performs steps of:
receiving a list of web sites, wherein the list of web sites is created based on a plurality of factors, and wherein the factors include more than one of newly registered domains, suspicious domains flagged by heuristic signatures, unclassified domains in a network security system, country-specific domains, and a targeted scan based on Content Management System (CMS);
anonymously browsing to each web site in the list;
receiving a response based on the browsing;
performing de-obfuscation of content in the response;
analyzing the content by performing a difference between the de-obfuscated content and obfuscated content of the response to identify JS obfuscation in the content;
classifying each web site as malicious or not based on a presence of any of a JS function and a domain in the de-obfuscated content not present in obfuscated content, wherein a web side is classified as malicious when the de-obfuscated content includes new functions or domains not present in the obfuscated content;
providing a blacklist of web sites classified as malicious; and
monitoring the blacklist to continually remove web sites that no longer exist or that no longer exhibit suspicious behavior.
2. The non-transitory computer-readable storage medium of claim 1 , wherein the computer-readable code stored is further configured to program the server to perform the step of
distributing the blacklist to a cloud-based system and user devices for local blocking of access at the user devices.
3. The non-transitory computer-readable storage medium of claim 1 , wherein the classifying further includes detection of hidden Inline Frames in the response.
4. The non-transitory computer-readable storage medium of claim 1 , wherein the computer-readable code stored is further configured to program the server to perform the step of
creating the list of web sites periodically based on a plurality of factors.
5. The non-transitory computer-readable storage medium of claim 1 , wherein the anonymously browsing utilizes a Virtual Private Network (VPN) to obscure the server.
6. The non-transitory computer-readable storage medium of claim 1 , wherein the computer-readable code stored is further configured to program the server to perform steps of
detecting an obfuscated coin miner in the content; and
classifying the web site as malicious based on the detected obfuscated coin miner.
7. A server comprising:
a network interface communicatively coupled to a network;
a processor communicatively coupled to the network interface; and
memory storing computer-executable instructions that, when executed, cause the processor to
receive a list of web sites receiving a list of web sites, wherein the list of web sites is created based on a plurality of factors, and wherein the factors include more than one of newly registered domains, suspicious domains flagged by heuristic signatures, unclassified domains in a network security system, country-specific domains, and a targeted scan based on Content Management System (CMS);
anonymously browse to each web site in the list;
receive a response based on the browsing;
perform de-obfuscation of content in the response;
analyze the content by performing a difference between the de-obfuscated content and obfuscated content of the response to identify JS obfuscation in the content;
classify each web site as malicious or not based on a presence of any of a JS function and a domain in the de-obfuscated content not present in obfuscated content, wherein a web site is classified as malicious when the de-obfuscated content includes new functions or domains not present in the obfuscated content;
provide a blacklist of web sites classified as malicious; and
monitor the blacklist to continually remove web sites that no longer exist or that no longer exhibit suspicious behavior.
8. The server of claim 7 , wherein the computer-readable code stored is further configured to program the server to perform the step of
distributing the blacklist to a cloud-based system and user devices for local blocking of access at the user devices.
9. The server of claim 7 , wherein the classifying further includes detection of hidden Inline Frames in the response.
10. The server of claim 7 , wherein the computer-executable instructions that, when executed, further cause the processor to
creating the list of web sites periodically based on a plurality of factors.
11. The server of claim 7 , wherein the anonymously browsing utilizes a Virtual Private Network (VPN) to obscure the server.
12. A method comprising:
receiving a list of web sites, wherein the list of web sites is created based on a plurality of factors, and wherein the factors include more than one of newly registered domains, suspicious domains flagged by heuristic signatures, unclassified domains in a network security system, country-specific domains, and a targeted scan based on Content Management System (CMS);
anonymously browsing to each web site in the list;
receiving a response based on the browsing;
performing de-obfuscation of content in the response;
analyzing the content by performing a difference between the de-obfuscated content and obfuscated content of the response to identify JS obfuscation in the content;
classifying each web site as malicious or not based on a presence of any of a JS function and a domain in the de-obfuscated content not present in obfuscated content, wherein a web site is classified as malicious when the de-obfuscated content includes new functions or domains not presented in the obfuscated content;
providing a blacklist of web sites classified as malicious; and
monitoring the blacklist to continually remove web sites that no longer exist or that no longer exhibit suspicious behavior.
13. The method of claim 12 , further comprising
distributing the blacklist to a cloud-based system and user devices for local blocking of access at the user devices.
14. The method of claim 12 , further comprising
creating the list of web sites periodically based on a plurality of factors.