Systems and methods of constructing a network topology
The technology disclosed relates to sub-clustering within service clusters in real-time. In particular, it relates to accessing a network topology that records node data and connection data including processes running on numerous hosts grouped into local services on the hosts, the local services running on multiple hosts grouped into service clusters and sub-clusters of service clusters, and network connections used by the service clusters to connect the hosts grouped into service connections, wherein the node data includes software versions of the processes and process data with configuration files and clustering the multiple hosts with the service clusters into the sub-clusters based at least in part on the software versions.
1. A method of real-time tracking network topology changes, the method including:
accessing from a network topology tracker a database that rolls up a network topology that includes the following components:
processes running on numerous hosts and grouped into local services on the hosts;
the local services running on multiple hosts and grouped into services; and
network connections used by the services to connect the hosts and grouped into service connections;
receiving at a network topology tracking queue, in real-time, filtered network feature data from data fetchers and processing the received data against the network topology, including:
queuing new processes in the received data that are not present in the network topology to be classified into services after a new process threshold is reached;
queuing new connections in the received data that are not present in the network topology to be classified into service connections after a new connection threshold is reached;
queuing new instances of services in the received data that are not present in the network topology to be added to the network topology after a new instance threshold is reached;
queuing new service connections in the received data that are not present in the network topology to be added to the network topology after a new service connection threshold is reached; and
matching processes, instances of services and service connections from the received data with the components of the network topology and updating a timestamp in the matching components;
periodically processing one or more queues from the received data to classify new processes into services after the new process threshold is reached and new connections into service connections after the new connection threshold is reached;
periodically processing by the network topology tracker one or more queues from the received data to add to the network topology new instances of services after the new instance threshold is reached and to add to the network topology new service connections after the new service connection threshold is reached;
periodically processing by the network topology tracker the network topology to remove out-of-date processes and self-depreciated processes, instances of services and service connections; and
persisting an updated network topology after adding and removing from the network topology.
2. The method of claim 1 , wherein the received data has been de-duplicated to reduce complementary views of a connection from opposite ends of the connection.
3. The method of claim 1 , further including:
queuing new hosts in the received data that are not present in the network topology to be classified into services after a new host threshold is reached; and
periodically processing one or more queues from the received data to classify new hosts into services after the new host threshold is reached.
4. The method of claim 1 , wherein the updated timestamp indicates when a process, instance of a service or a service connection was last seen and/or when a process, instance of a service or a service connection will expire.
5. The method of claim 1 , further including:
queuing new apps of multiple services in the received data that are not present in the network topology to be added to the network topology after a new app threshold is reached; and
periodically processing by the network topology tracker one or more queues from the received data to add new apps to the network topology after the new app threshold is reached.
6. The method of claim 1 , wherein at least one of the new connections includes a process endpoint on a host that does not report network feature data for network topology processing.
7. The method of claim 6 , further including:
classifying the process endpoint and the host as an external process; and
adding the external process to the network topology.
8. The method of claim 1 , wherein the new process threshold and the new connection threshold specify at least one of:
one or more time periods to pass before processing the new processes and the new connections from the queues; and
one or more periodic appearance criteria to satisfy before processing the new processes and the new connections from the queues.
9. A non-transitory computer readable storage medium impressed with computer program instructions, the instructions, when executed on a processor, implement the method of claim 1 .
10. A system including one or more processors coupled to memory, the memory loaded with computer instructions, the instructions, when executed on the processors, implement actions of claim 1 .