Method and system for identifying a preferred set of hierarchically structured items in streaming data
A computer implemented method and system for identifying a preferred set of hierarchically structured items in streaming data for analyzing Netflow data to identify those network destinations that are currently the target of a DDoS attack and to automatically select a set of network prefixes such that diversion routes for the prefixes are sent to the routers to divert attack traffic to TMS devices, The method includes searching sets of Hierarchical Heavy Hitters wherein each set corresponds to a different fraction of a total volume of network traffic and scoring each set according to an arbitrary scoring function. A certain set is selected and scored with a ‘good’ score and a member of the ‘good’ scored set is ranked in accordance with an arbitrary ranking function. A subset of the ‘good’ scored set is selected such that the volume associated with the subset is in close proximity to a user-specified total whereby the selected subset becomes a set of recommended prefixes.
1. A method for identifying a preferred set of hierarchically structured items in streaming data for analyzing network data traffic (Netflow data) to identify those network destinations that are currently the target of a Distributed Denial of Service (DDoS) attack and to automatically select a set of network prefixes such that diversion routes for the prefixes are sent to routers to divert attack traffic to Threat Mitigation Systems (TMS) devices, comprising:
searching sets of Hierarchical Heavy Hitters (H3PREF) wherein each set corresponds to a different fraction of a total volume of network traffic and includes at least: 1) a lattice structure modeling hierarchy in an ID space; 2) a counter update procedure; and 3) a stream merging procedure;
scoring each set according to an arbitrary scoring function that includes a mix of network mask lengths present in the set and a number of members in the set;
selecting a good scored set with a value indicative that a network device is not associated with a DDoS attack;
ranking member of the good scored set in accordance with an arbitrary ranking function; and
selecting a subset of the good scored set such that a volume associated with the subset is in close proximity to a user-specified total whereby the selected subset becomes a set of recommended prefixes.
2. A computer system for identifying a preferred set of hierarchically structured items in streaming data for analyzing network data traffic (Netflow data) to identify those network destinations that are currently the target of a Distributed Denial of Service (DDoS) attack and to automatically select a set of network prefixes such that diversion routes for the prefixes are sent to routers to divert attack traffic to Threat Mitigation Systems (TMS) devices, comprising:
a memory configured to store instructions;
a processor disposed in communication with said memory, wherein said processor upon execution of the instructions is configured to:
search sets of Hierarchical Heavy Hitters (H3PREF) wherein each set corresponds to a different fraction of a total volume of network traffic and includes at least: 1) a lattice structure modeling hierarchy in an ID space; 2) a counter update procedure; and 3) a stream merging procedure;
score each set according to an arbitrary scoring function that includes a mix of network mask lengths present in the set and a number of members in the set;
select a good scored set scored with a value indicative that a network device is not associated with a DDoS attack;
rank member of the good scored set in accordance with an arbitrary ranking function; and
select a subset of the good scored set such that a volume associated with the subset is in close proximity to a user-specified total whereby the selected subset becomes a set of recommended prefixes.
3. The computer method as recited in claim 1 , wherein each H3PREF further includes a selection process for identifying a set of IDs representative of a user-specific fraction of a total volume of the streaming data.
4. The computer system as recited in claim 2 , wherein each H3PREF further includes a selection process for identifying a set of IDs representative of a user-specific fraction of a total volume of the streaming data.