Flow logging for connection-based anomaly detection
View Patent ↗A plurality of flow collector devices is disposed to collect flow information on a network. Duplicate flow records received from the flow collectors are eliminated by determining whether a pair of flow records has the same, source and destination flow identifiers and were received within a predefined time-period. Non-duplicated flow records received from the plurality of flow collector devices are stored and used to produces a connection table that maps each node on the network to a record that stores information about traffic to or from the node from non-duplicated flow records. The connection table stores statistical information of packets on the network based on a time-slice basis.
1. A computer implemented method, comprising:
receiving, at an aggregator, flow records from a plurality of flow collector devices that are disposed in a network to collect flow information;
prior to storing the flow records in the aggregator,
in response to determining that a first flow record and a second flow record, received at different flow collectors within a predefined time-period, have the same source and destination flow identifiers, and that the number of bytes, packets, and connections in the first flow record is greater than the number of bytes, packets, and connections, respectively, in the second flow record,
selecting the first flow record for storing in the aggregator, and
discarding the second flow record;
storing non-discarded flow records in the aggregator; and
creating, at the aggregator, a connection table comprising the stored flow records indexed by source and destination flow identifiers.
2. The method of claim 1 , wherein the connection table maps each node on the network to a record that stores information about traffic to or from the node.
3. The method of claim 1 , wherein the connection table stores statistical information of packets on the network based on a time-slice basis.
4. The method of claim 1 , wherein the predefined time-period is a time slice.
5. The method of claim 1 , wherein determining that two flows have the same source and destination flow identifiers includes determining that the two flows have the same source and destination addresses.
6. The method of claim 1 , wherein determining that two flows have the same source and destination flow identifiers includes determining that the two flows have the same source and destination ports.
7. The method of claim 1 , wherein determining that two flows have the same source and destination flow identifiers includes determining that the two flows have the same protocol.
8. The method of claim 1 , wherein the source and destination flow identifiers include the source and destination addresses or ports.
9. The method of claim 1 wherein the source and destination statistics include the source and destination addresses, source and destination ports, and protocol.
10. The method of claim 1 , wherein the flow records in the connection table are indexed by time.
11. The method of claim 1 , wherein the flow records in the connection table are indexed by source address, destination address and time.
12. A system comprising:
a computing device including a non-transitory computer readable medium storing a computer program that includes instructions to cause the computing device to:
receive, at an aggregator, flow records from a plurality of flow collector devices that are disposed in a network to collect flow information;
prior to storing the flow records in the aggregator,
in response to determining that a first flow record and a second flow record, received at different flow collectors within a predefined time-period, have the same source and destination flow identifiers, and that the number of bytes, packets, and connections in the first flow record is greater than the number of bytes, packets, and connections, respectively, in the second flow record,
select the first flow record for storing in the aggregator, and
discard the second flow record;
store non-discarded flow records in the aggregator; and
create a connection table comprising the stored flow records indexed by source and destination flow identifiers.
13. The system of claim 12 , wherein the connection table maps each node on the network to a record that stores information about traffic to or from the node.
14. The system of claim 12 , wherein the connection table stores statistical information of packets on the network based on a time-slice basis.
15. The system of claim 12 wherein the time-period is a time slice.
16. The system of claim 12 , wherein the instructions to determine that two flows have the same source and destination flow identifiers include instructions to determine that the two flows have the same source and destination addresses, source and destination ports, and protocol.
17. A non-transitory computer readable storage medium storing an executable that includes instructions to cause a computing device to:
receive, at an aggregator, flow records from a plurality of flow collector devices that are disposed in a network to collect flow information;
prior to storing the flow records in the aggregator,
in response to determining that a first flow record and a second flow record, received at different flow collectors within a predefined time-period, have the same source and destination flow identifiers, and that the number of bytes, packets, and connections in the first flow record is greater than the number of bytes, packets, and connections, respectively, in the second flow record,
select the first flow record for storing in the aggregator, and
discard the second flow record;
store non-discarded flow records in the aggregator; and
create a connection table comprising the stored flow records indexed by source and destination identifiers.
18. The non-transitory computer readable storage medium of claim 17 , wherein the connection table maps each node on the network to a record that stores information about traffic to or from the node.
19. The non-transitory computer readable storage medium of claim 17 , wherein the connection table stores statistical information of packets on the network based on a time-slice basis.
20. The non-transitory computer readable storage medium of claim 17 , wherein the predefined time-period is a time slice.
21. The non-transitory computer readable storage medium of claim 17 , wherein the instructions to determine that two flows have the same source and destination flow identifiers include instructions to determine whether the two flows have the same source and destination addresses, source and destination ports, and protocol.
22. A computer implemented method, comprising:
receiving, at an aggregator, flow records from a plurality of flow collector devices that are disposed in a network to collect flow information;
prior to storing the flow records in the aggregator,
in response to determining that two or more flow records, received at different flow collectors within a predefined time-period, have the same source and destination flow identifiers,
selecting a flow record from the two or more flow records that has the maximum number of bytes, packets, and connections among the two or more flow records, and
discarding flow records that were not selected;
storing non-discarded flow records in the aggregator; and
creating, at the aggregator, a connection table comprising the stored flow records indexed by source and destination flow identifiers, wherein the connection table maps each node on the network to a record that stores information about traffic to or from the node.
23. The method of claim 22 , wherein the connection table stores statistical information of packets on the network based on a time-slice basis.
24. The method of claim 22 , wherein the predefined time-period is a time slice.
25. The method of claim 22 , wherein determining that two flows have the same source and destination flow identifiers includes determining that the two flows have the same source and destination addresses, the same source and destination ports, and the same protocol.
26. The method of claim 22 , wherein the flow records in the connection table are indexed by time.
27. The method of claim 22 , wherein the flow records in the connection table are indexed by source address, destination address and time.
28. A system comprising:
a computing device including a computer readable medium storing a computer program that includes instructions to cause the computing device to:
receive, at an aggregator, flow records from a plurality of flow collector devices that are disposed in a network to collect flow information;
prior to storing the flow records in the aggregator,
in response to determining that two or more flow records, received at different flow collectors within a predefined time-period, have the same source and destination flow identifiers,
select a flow record from the two or more flow records that has the maximum number of bytes, packets, and connections among the two flow records, and
discard flow records from the two or more flow records that were not selected;
store non-discarded flow records in the aggregator; and
create a connection table comprising the stored flow records indexed by source and destination flow identifiers, wherein the connection table maps each node on the network to a record that stores information about traffic to or from the node.
29. The system of claim 28 wherein the connection table stores statistical information of packets on the network based on a time-slice basis.
30. The system of claim 28 wherein instructions to determine that two flows have the same source and destination flow identifiers include instructions to determine whether the two flows have the same source and destination hosts, source and destination ports and protocol.
31. A non-transitory computer readable storage device storing executable instructions to cause a computing device to:
receive, at an aggregator, flow records from a plurality of flow collector devices that are disposed in a network to collect flow information;
prior to storing the flow records in the aggregator,
in response to determining that two or more flow records, received at different flow collectors within a predefined time-period, have the same source and destination flow identifiers,
select a flow record from the two flow records that has the maximum number of bytes, packets, or connections among the two or more flow records, and
discard flow records from the two or more flow records that were not selected;
store non-discarded flow records in the aggregator; and
create a connection table comprising the stored flow records indexed by source and destination flow identifiers, wherein the connection table maps each node on the network to a record that stores information about traffic to or from the node.
32. The computer program product of claim 31 , wherein the connection table stores statistical information of packets on the network based on a time-slice basis.
33. The computer program product of claim 31 , wherein the instructions to determine that two flows have the same source and destination flow identifiers include instructions to determine that the two flows have the same source and destination addresses, source and destination ports and protocol.