Methods and systems for distribution and retrieval of network traffic records
A method includes transmitting, by a distribution server, to each of a plurality of worker computers, a request for an enumeration of Internet Protocol (IP) addresses ranked according to a criterion. The method includes receiving, by the distribution computer, from a first of the plurality of worker computers, a first partial enumeration of the requested IP addresses ranked according to the criterion, the first partial enumeration stored in a hash table. The method includes receiving, by the distribution computer, from a second of the plurality of worker computers, a second partial enumeration of the requested IP addresses ranked according to the criterion, the second partial enumeration stored in a hash table. The method includes generating, by the distribution computer, a combined enumeration including the first partial enumeration and the second partial enumeration, the combined enumeration ranked according to the criterion. The distribution computer deduplicates the combined enumeration.
1. A method for generating a combined, deduplicated enumeration of network traffic records received from a plurality of worker computers in a computer network, and providing a network security assessment based on the combined, deduplicated enumeration of network traffic records, the method performed by at least one computer processor executing computer program instructions stored on at least one non-transitory computer-readable medium, the method comprising:
transmitting, by a distribution computer in the computer network, to each of the plurality of worker computers in the computer network, a request for an enumeration of Internet Protocol (IP) addresses ranked according to a criterion;
receiving, by the distribution computer, from a first of the plurality of worker computers, a first partial enumeration of the IP addresses ranked according to the criterion, the first partial enumeration stored in a hash table;
receiving, by the distribution computer, from a second of the plurality of worker computers, a second partial enumeration of the IP addresses ranked according to the criterion, the second partial enumeration stored in a hash table;
generating, by the distribution computer, a combined enumeration including the first partial enumeration and the second partial enumeration, the combined enumeration ranked according to the criterion;
deduplicating, by the distribution computer, the combined enumeration;
identifying, by the distribution computer, an abnormal communications pattern in the combined, deduplicated enumeration of network traffic records;
generating, by the distribution computer, a behavioral fingerprint based upon the identifying of the abnormal communications pattern; and
providing, by the distribution computer, a network security assessment based on the combined, deduplicated enumeration of network traffic records and the behavioral fingerprint.
2. The method of claim 1 , wherein said transmitting further comprises automatically transmitting the request at predetermined time intervals.
3. The method of claim 1 , wherein said transmitting further comprises transmitting the request for the enumeration of IP addresses associated with a protocol identified in each of a plurality of network traffic records, the enumeration of the IP addresses ranked according to the criterion.
4. The method of claim 1 , wherein said transmitting further comprises transmitting the request for the enumeration of IP addresses associated with a port identified in each of a plurality of network traffic records, the enumeration of the IP addresses ranked according to the criterion.
5. The method of claim 1 , wherein said transmitting the request for the enumeration of IP addresses further comprises transmitting a request for an enumeration of IP addresses identified as a destination address in each of a plurality of network traffic records, the enumeration of the IP addresses ranked according to the criterion.
6. The method of claim 1 , wherein said transmitting the request for the enumeration of IP addresses further comprises transmitting a request for an enumeration of IP addresses identified as a source address in each of a plurality of network traffic records, the enumeration of the IP addresses ranked according to the criterion.
7. The method of claim 1 , wherein said receiving, by the distribution computer, from the first of the plurality of worker computers, the first partial enumeration of the IP addresses further comprises receiving, by the distribution computer, a result of a search, by the first of the plurality of worker computers, for at least one IP address satisfying the criterion.
8. The method of claim 1 , wherein said receiving, by the distribution computer, from the second of the plurality of worker computers, the second partial enumeration of the IP addresses further comprises receiving, by the distribution computer, a result of a search, by the second of the plurality of worker computers, for at least one IP address satisfying the criterion.
9. The method of claim 1 , wherein said deduplicating further comprises performing hash table merging to automatically deduplicate the first partial enumeration and the second partial enumeration.
10. A non-transitory computer readable medium comprising computer program instructions tangibly stored on the non-transitory computer readable medium, wherein the computer program instructions are executable by at least one computer processor to perform a method for generating a combined, deduplicated enumeration of network traffic records received from a plurality of worker computers, and providing a network security assessment based on the combined, deduplicated enumeration of the network traffic records, the method comprising:
transmitting, by a distribution computer in a computer network, to each of a plurality of worker computers in the computer network, a request for an enumeration of Internet Protocol (IP) addresses ranked according to a criterion;
receiving, by the distribution computer, from a first of the plurality of worker computers, a first partial enumeration of the IP addresses ranked according to the criterion, the first partial enumeration stored in a hash table;
receiving, by the distribution computer, from a second of the plurality of worker computers, a second partial enumeration of the IP addresses ranked according to the criterion, the second partial enumeration stored in a hash table;
generating, by the distribution computer, a combined enumeration including the first partial enumeration and the second partial enumeration, the combined enumeration ranked according to the criterion;
deduplicating, by the distribution computer, the combined enumeration
identifying, by the distribution computer, an abnormal communications pattern in the combined, deduplicated enumeration of network traffic records;
generating, by the distribution computer, a behavioral fingerprint based upon the identifying of the abnormal communications pattern; and
providing, by the distribution computer, a network security assessment based on the combined, deduplicated enumeration of network traffic records and the behavioral fingerprint.
11. The non-transitory computer readable medium of claim 10 , wherein said transmitting further comprises automatically transmitting the request at predetermined time intervals.
12. The non-transitory computer readable medium of claim 10 , wherein said transmitting further comprises transmitting the request for the enumeration of IP addresses associated with a protocol identified in each of a plurality of network traffic records, the enumeration of the IP addresses ranked according to the criterion.
13. The non-transitory computer readable medium of claim 10 , wherein said transmitting further comprises transmitting the request for the enumeration of IP addresses associated with a port identified in each of a plurality of network traffic records, the enumeration of the IP addresses ranked according to the criterion.
14. The non-transitory computer readable medium of claim 10 , wherein said deduplicating further comprises performing hash table merging to automatically deduplicate the first partial enumeration and the second partial enumeration.