Stackable aggregation for connection based anomaly detection
A system includes a plurality of collector devices that are disposed to collect statistical information on packets that are sent between nodes on a network. The system also includes a stackable aggregator that receives network data from the plurality of collector devices, and which produces a connection table that maps each node on the network to a record that stores information about traffic to or from the node. The stackable aggregator includes a manager blade, a database blade, and two or more, analyzer blades.
1. A system, comprising:
a plurality of collector devices that are disposed to collect statistical information on packets sent between nodes on a network;
a stackable aggregator that receives network data from the plurality of collector devices, the aggregator producing a connection table that maps each node on the network to a record that stores information about traffic to or from the node, the stackable aggregator comprising:
a manager blade,
a database blade, and
two or more analyzer blades.
2. The system of claim 1 wherein the manager blade includes an event manager to correlate and report events to an operator console.
3. The system of claim 2 wherein each analyzer blade is responsible for storing and analyzing approximately 1/N of network data, where N corresponds to number of analyzer blades in the aggregator.
4. The system of claim 1 wherein one of the analyzer blades, comprises:
a dispatcher that receives flow records and traffic counters from network sensors and forwards flow records and statistical data on network traffic to a specific one of the two or more analyzer blades.
5. The system of claim 4 wherein the dispatcher produces a hash of source and destination host identification values in the flow records or statistic records received, and uses the hash of the source and destination host identification values to distribute the flow records or statistic records to particular analyzer blades.
6. The system of claim 1 wherein each of the analyzer blades comprises:
local storage for storing flow records.
7. The system of claim 1 wherein each of the analyzer blades produces statistical data for its fraction of the network traffic.
8. The system of claim 2 wherein each of the analyzer blades examines statistical data to determine the presence of anomalies, and as anomalies are determined by the analyzer blades, data regarding the anomalies are forwarded to the event manager.
9. The system of claim 1 wherein each of the analyzer blades receives flow records from a dispatcher in the one of the analyzer blades comprising a dispatcher process.
10. The system of claim 1 wherein the database blade manages a database that stores the connection table.
11. The system of claim 10 wherein the connection table includes a plurality of records indexed by source address, destination address and time.
12. The system of claim 11 wherein the connection table includes a plurality of connection sub-tables to track data at different time scales.
13. The system of claim 1 wherein each analyzer blade of the aggregator includes:
at least two processors; and
memory associated with the at least two processors.
14. A method, comprises:
collecting statistical information on packets that are sent between nodes on a network;
dispatching statistical information via a reliable protocol to one of two or more analyzer blades in an aggregator to produce a connection table that maps each node on the network to a record that stores information about traffic to or from the node wherein the aggregator comprises a manager blade, a database blade, and the two or more analyzer blades.
15. The method of claim 14 wherein each analyzer blade is responsible for storing and analyzing approximately 1/N of network data, where N corresponds to a number of analyzer blades in the aggregator.
16. The method of claim 14 further comprising:
assembling flow records and traffic counters from network sensors; and
forwarding flow records and statistical data on network traffic to a specific one of two or more analyzer blades to assemble the connection table.
17. A non-transitory computer-readable storage device storing instructions that when executed by a computer cause the computer to:
receive network data from a plurality of collector devices that collect statistical information on packets that are sent between nodes on a network; and
dispatch received network data from a plurality of collector devices via a reliable protocol to a specific one of the two or more analyzer blades, in an aggregator to produce multiple connection tables each table storing a portion of the collect statistical information on packets sent on the network to a record wherein the aggregator comprises a manager blade, a database blade, and the two or more analyzer blades.
18. The storage device of claim 17 further comprising instructions cause a computer to: correlate and report events to an operator console.
19. The storage device of claim 17 further comprising instructions to cause a computer to:
hash source and destination host identification values in the flow records or statistic records received; and
distribute the flow records or statistic records to particular analyzer blades according to the hash of the source and destination host identification values.
20. The storage device of claim 18 further comprising instructions to cause a computer to:
examine statistical data to determine the presence of anomalies; and
forward anomalies to an event manager process.