Identifying significant behaviors within network traffic

1. A computer-implemented method for identifying significant clusters from traffic traversing a communications link, said method comprising:

accessing said communication link with a system including a device configured to read headers of a packet;

observing a plurality of flows traversing said communication link with said device; grouping together one or more of said plurality of flows into clusters based on said flows having at least one of the same value for the source IP address (srcIP), destination IP address (dstIP), source port (srcPrt), or destination port (dstPrt),

assigning a probability value to each of said clusters, wherein said probability value relates to a cluster property;

placing each of said clusters into a set of clusters;

selecting a probability threshold and an uncertainty threshold;

removing one or more of said clusters whose assigned probability value is above said probability threshold, wherein each of the one or more removed clusters is identified as a significant cluster;

computing a relative uncertainty value indicating a level of variability or uniformity among the probability values assigned to the clusters that remain in said set of clusters;

repeating a series of steps until said relative uncertainty value is equal to or exceeds said uncertainty threshold, wherein said series of steps includes:

(1) decreasing said probability threshold,

(2) removing one or more of said clusters remaining in said set of cluster whose assigned probability value is above said probability threshold,

(3) identifying each of the one or more removed clusters as a significant cluster,

(4) re-computing a relative uncertainty value indicating a level of variability or uniformity among the probability values assigned to the clusters that remain in said set of clusters, and

(5) comparing said relative uncertainty value to said uncertainty threshold; and

assigning said significant clusters to one of a plurality of behavior classes; and

generating a profile characterizing a plurality of traffic patterns, wherein said profile associates one or more of said plurality of behavior classes with each of said plurality of traffic patterns and said plurality of traffic patterns includes one or more profiles associated with malicious behavior or with exploit behavior.

2. The method of claim 1 , wherein said communications link is a backbone of a transit network.

3. The method of claim 1 , wherein said cluster property indicates a volume of flows in a cluster.

4. The method of claim 1 , wherein each probability value associated with said one or more removed clusters is larger than each probability value associated with the clusters remaining in said set of clusters.

5. The method of claim 1 , wherein said uncertainty threshold is equal to or greater than 0.9.

6. A computer-implemented method for identifying a set of significant clusters from traffic on a computer network, the method comprising:

(a) obtaining a source IP address (srcIP), a destination IP address (dstIP), a source port (srcPrt), or a destination port (dstPrt) associated with each of a plurality of flows traversing a link on said computer network via a device configured to read headers of a packet;

(b) creating clusters by grouping together one or more of said plurality of flows having the same source IP address (srcIP), destination IP address (dstIP), source port (srcPrt), or destination port (dstPrt), wherein each of said clusters is placed into a set of clusters;

(c) assigning a probability value to each of said clusters, wherein said probability value relates to the number of flows in a cluster;

(d) removing from said set of clusters one or more clusters having an assigned probability value above a probability threshold;

(e) associating each of the one or more removed cluster with a set of significant clusters;

(f) decreasing said probability threshold;

(g) determining a relative uncertainty value that indicates a level of relative uncertainty or uniformity among the probability values assigned to the clusters in said set of clusters;

(h) repeating said steps (d)-(g) until said uncertainty value equals or exceeds an uncertainty threshold, wherein said uncertainty threshold is equal to or greater than 0.75; and

dividing said set of significant clusters into a plurality of behavior classes by identifying one or more behavioral characteristics for each of said plurality of significant clusters, wherein said one or more behavioral characteristics includes malicious or exploit behavior.

7. The method of claim 6 , wherein said decreasing said probability threshold includes reducing said probability threshold by an exponentially decreasing factor.