Techniques for determining threat intelligence for network infrastructure analysis
Embodiments of the present disclosure are directed to a network analytic system for tracking and analysis of network infrastructure for network-based digital assets. The network analytic system can detect and track a relationship between assets based on one or more attributes related or shared between any given assets. The network analytic system can analyze network-based digital assets to determine information about a website (e.g., information about electronic documents, such as web pages) that has be used to detect phishing and other abuse of the website. The network analytic system can analyze data about network-based assets to determine whether any are being used or connected to use of unauthorized or malicious activity or known network-based assets. Based on the relationship identified, the network analytic system can associate or link assets together. The network analytic system may provide an interface to view data sets generated by the network analytic system.
1. A computer-implemented method comprising, at a computer system:
accepting a crawl configuration, which specifies a sequence of internet-facing assets to be monitored, based on verification that the crawl configuration is free of errors;
selecting a proxy server via which to access an internet-facing asset, which is included in the sequence of internet-facing assets, based on the crawl configuration, the proxy server enabling a bot to anonymously interact with the internet-facing asset;
selecting the bot, which obtains data from the internet-facing asset by accessing the internet-facing asset anonymously via the proxy server, based on the crawl configuration;
monitoring the internet-facing asset according to the crawl configuration, wherein the monitoring comprises identifying an event that is indicated by the data, which is obtained by the bot, by analyzing the data;
based on the event being identified, causing data related to the event to be displayed on a display device;
analyzing network data associated with the event; and
causing information related to the network data to be displayed on the display device.
2. The computer-implemented method of claim 1 , wherein accepting the crawl configuration comprises:
accepting the crawl configuration based on verification that the crawl configuration includes requisite username and password information.
3. The computer-implemented method of claim 1 , wherein the crawl configuration indicates an order in which the internet-facing assets are to be monitored, the order based on how close links to the internet-facing assets are to a root directory of a web page.
4. The computer-implemented method of claim 1 , wherein the proxy server simulates access of the internet-facing asset by devices from different geographic locations.
5. The computer-implemented method of claim 1 , wherein the proxy server simulates access of the internet-facing asset by different types of web browsers.
6. The computer-implemented method of claim 1 , wherein selecting the proxy server comprises:
selecting the proxy server from a plurality of proxy servers based on the crawl configuration, which specifies the sequence of the internet-facing assets to be monitored.
7. The computer-implemented method of claim 1 , wherein selecting the bot comprises:
selecting the bot from a plurality of bots based on the crawl configuration, which specifies the sequence of the internet-facing assets to be monitored.
8. The computer-implemented method of claim 1 , wherein monitoring the internet-facing asset comprises:
monitoring the internet-facing asset according to a rule that is defined based on a user-specified keyword.
9. The computer-implemented method of claim 1 , wherein analyzing the network data comprises:
analyzing the network data, which indicates Internet activity associated with the event.
10. A system comprising:
memory; and
a processing system coupled to the memory, the processing system configured to:
select a proxy server via which to access an internet-facing asset based on a crawl configuration that indicates an order in which a plurality of internet-facing assets, which includes the internet-facing asset, are to be monitored, the order based on how close a plurality of respective links to the plurality of internet-facing assets are to a root directory of a web page, the proxy server enabling a bot to anonymously interact with the internet-facing asset;
based on the crawl configuration, select the bot, which obtains data from the internet-facing asset by accessing the internet-facing asset anonymously via the proxy server;
monitor the internet-facing asset according to the crawl configuration, wherein the monitoring comprises identifying an event that is indicated by the data, which is obtained by the bot, by analyzing the data;
based on the event being identified, cause data related to the event to be displayed on a display device;
analyze network data associated with the event; and
cause information related to the network data to be displayed on the display device.
11. The system of claim 10 , wherein the processing system is further configured to: accept the crawl configuration based on verification that the crawl configuration is free of errors.
12. The system of claim 11 , wherein the processing system is configured to:
accept the crawl configuration based on verification that the crawl configuration includes requisite username and password information.
13. The system of claim 10 , wherein the proxy server simulates access of the internet-facing asset by devices from different geographic locations.
14. The system of claim 10 , wherein the proxy server simulates access of the internet-facing asset by different types of web browsers.
15. The system of claim 10 , wherein the processing system is configured to:
select the proxy server from a plurality of proxy servers based on the crawl configuration, which indicates the order in which the plurality of internet-facing assets are to be monitored.
16. The system of claim 10 , wherein the processing system is configured to:
select the bot from a plurality of bots based on the crawl configuration, which indicates the order in which the plurality of internet-facing assets are to be monitored.
17. The system of claim 10 , wherein the processing system is configured to:
monitor the internet-facing asset according to a rule that is defined based on a user-specified keyword.
18. The system of claim 10 , wherein the processing system is configured to:
analyze the network data, which indicates Internet activity associated with the event.
19. A non-transitory computer-readable storage medium storing instructions that, when executed, cause a processor-based system to perform operations, the operations comprising:
accept a crawl configuration, which specifies a sequence of internet-facing assets to be monitored, based on verification that the crawl configuration is free of errors;
select a proxy server via which to access an internet-facing asset, which is included in the sequence of internet-facing assets, based on the crawl configuration, the proxy server enabling a bot to anonymously interact with the internet-facing asset;
select the bot, which is configured to obtain data from the internet-facing asset by accessing the internet-facing asset anonymously via the proxy server, based on the crawl configuration;
monitor the internet-facing asset according to the crawl configuration, wherein the monitoring comprises identifying an event that is indicated by the data, which is obtained by the bot, by analyzing the data;
based on the event being identified, cause data related to the event to be displayed on a display device;
analyze network data associated with the event; and
cause information related to the network data to be displayed on the display device.
20. The non-transitory computer-readable storage medium of claim 19 , wherein the crawl configuration indicates an order in which the internet-facing assets are to be monitored, the order based on how close links to the internet-facing assets are to a root directory of a web page.