Method and system for detecting threats using metadata vectors
An approach for detecting network attacks using metadata vectors may initially involve receiving network communications or packets, extracting metadata items from the packets. The metadata items describe the communications without requiring deep content inspection of the data payload or contents. The communications may be clustered into groups using the metadata items. If a cluster exceeds a threshold, an alarm may be generated.
1. A method for detecting threats in networks using metadata, comprising:
receiving one or more network communications by tapping a network device that routes network communications between a single host and multiple hosts, wherein a network communication comprises at least a first communication in a first direction from the single host to the multiple hosts and a second communication in a second direction from the multiple hosts to the single host, the first communication corresponding to the single host sending copies of a payload to the multiple hosts, the second communication corresponding to response messages sent by the multiple hosts initiated in response to receiving copies of the payload from the single host, the second direction being opposite of the first direction;
extracting one or more metadata instances from the network communications that describe the network communications without inspecting contents of the network communications,
grouping the metadata instances into one or more clusters based on whether the metadata instances match; and
generating alert data for a cluster when an amount of metadata instances in the cluster exceeds a threshold.
2. The method of claim 1 , wherein the clusters correspond to a sliding window in which metadata instances older than a time duration are removed.
3. The method of claim 1 , wherein network communications are grouped into clusters by matching a plurality of metadata items in a first metadata instance to a plurality of metadata items in a second metadata instance.
4. The method of claim 3 , wherein the metadata items comprise at least one or more of the following group: a first communication identifier, a second communication identifier, a destination port data item, a first communication size data item, a second communication size data item, a time elapsed data item, or a non-zero data volume data item.
5. The method of claim 4 , wherein two metadata instances are grouped into a same cluster if one or more metadata items belonging to each of the two metadata instances strictly match.
6. The method of claim 4 , wherein two metadata instances are grouped into a same cluster if one or more metadata items belonging to each of the two metadata instances are within a specified range or percentage threshold.
7. The method of claim 1 , further comprising a training window in which network communications are grouped into clusters.
8. The method of claim 4 , wherein the first communication identifier corresponds to a number of starting bytes of the first communication, and the second communication identifier corresponds to a number of starting bytes of the second communication.
9. A system for detecting threats in networks using metadata, comprising:
a computer processor to execute a set of program code instructions;
a memory to hold the set of program code instructions, in which the set of program code instructions comprises program code to perform:
receiving one or more network communications by tapping a network device that routes network communications between a single host and multiple hosts, wherein a network communication comprises at least a first communication in a first direction from the single host to the multiple hosts and a second communication in a second direction from the multiple hosts to the single host, the first communication corresponding to the single host sending copies of a payload to the multiple hosts, the second communication corresponding to response messages sent by the multiple hosts initiated in response to receiving copies of the payload from the single host, the second direction being opposite of the first direction;
extracting one or more metadata instances from the network communications that describe the network communications without inspecting contents of the network communications,
grouping the metadata instances into one or more clusters based on whether the metadata instances match; and
generating alert data for a cluster when an amount of metadata instances in the cluster exceeds a threshold.
10. The system of claim 9 , wherein the clusters correspond to a sliding window in which metadata instances older than a time duration are removed.
11. The system of claim 9 , wherein network communications are grouped into clusters by matching a plurality of metadata items.
12. The system of claim 11 , wherein the metadata items comprise at least one or more of the following group: a first communication identifier, a second communication identifier, a destination port data item, a first communication size data item, a second communication size data item, a time elapsed data item, or a non-zero data volume data item.
13. The system of claim 12 , wherein two metadata instances are grouped into a same cluster if one or more metadata items belonging to each of the two metadata instances strictly match.
14. The system of claim 12 , wherein two metadata instances are grouped into a same cluster if one or more metadata items belonging to each of the two metadata instances are within a specified range or percentage threshold.
15. The system of claim 9 , further comprising a training window in which network communications are grouped into clusters.
16. The system of claim 12 , wherein the first communication identifier corresponds to a number of starting bytes of the first communication, and the second communication identifier corresponds to a number of starting bytes of the second communication.
17. A computer program product embodied on a non-transitory computer readable medium, the non-transitory computer readable medium having stored thereon a sequence of instructions which, when executed by a processor causes the processor to execute a method detecting threats, the method comprising:
receiving one or more network communications by tapping a network device that routes network communications between a single host and multiple hosts, wherein a network communication comprises at least a first communication in a first direction from the single host to the multiple hosts and a second communication in a second direction from the multiple hosts to the single host, the first communication corresponding to the single host sending copies of a payload to the multiple hosts, the second communication corresponding to response messages sent by the multiple hosts initiated in response to receiving copies of the payload from the single host, the second direction being opposite of the first direction;
extracting one or more metadata instances from the network communications that describe the network communications without inspecting contents of the network communications;
grouping the metadata instances into one or more clusters based on whether the metadata instances match; and
generating alert data for a cluster when an amount of metadata instances in the cluster exceeds a threshold.
18. The computer program product of claim 17 , wherein the clusters correspond to a sliding window in which metadata instances older than a time duration are removed.
19. The computer program product of claim 17 , wherein network communications are grouped into clusters by matching a plurality metadata items.
20. The computer program product of claim 19 , wherein the metadata items comprise at least one or more of the following group: a first communication identifier, a second communication identifier, a destination port data item, a first communication size data item, a second communication size data item, a time elapsed data item, or a non-zero data volume data item.
21. The computer program product of claim 20 , wherein two metadata instances are grouped into a same cluster if one or more metadata items belonging to each of the two metadata instances strictly match.
22. The computer program product of claim 20 , wherein two metadata instances are grouped into a same cluster if one or more metadata items belonging to each of the two metadata instances are within a specified range or percentage threshold.
23. The computer program product of claim 17 , further comprising a training window in which network communications are grouped into clusters.
24. The computer program product of claim 20 , wherein the first communication identifier corresponds to a number of starting bytes of the first communication, and the second communication identifier corresponds to a number of starting bytes of the second communication.