Methods and systems for distribution and retrieval of network traffic records
View Patent ↗A method includes receiving, by a distribution server, a plurality of network traffic records. The distribution server generates a first hash from a first plurality of fields in a first of the plurality of network traffic records and generating a second hash from a second plurality of fields in a second of the plurality of network traffic records. The distribution server determines that the first and second of the plurality of network traffic records relate to a session. The method includes transmitting the first and second of the plurality of network traffic records to one of a plurality of worker computing devices selected based on the determination and on at least one of the first and second hash. The distribution server transmits a request for an enumeration of addresses ranked according to a criterion, receives partial enumerations from a plurality of worker computers, and generates a combined, deduplicated enumeration.
1. A method for generating a combined, deduplicated enumeration of network traffic records received from a plurality of worker computers and providing a network security assessment based on the enumeration, the method performed by at least one computer processor executing computer program instructions stored on at least one non-transitory computer-readable medium, the method comprising:
receiving, by a distribution server in a computer network, from an exporter device, a plurality of network traffic records;
generating, by the distribution server, a first hash from a first plurality of fields in a first of the plurality of network traffic records;
generating, by the distribution server, a second hash from a second plurality of fields in a second of the plurality of network traffic records;
comparing, by the distribution server, the first hash and the second hash;
determining, by the distribution server, that the first of the plurality of network traffic records and the second of the plurality of traffic records relate to a session, based upon the comparison;
transmitting, by the distributions server, the first of the plurality of network traffic records and the second of the plurality of traffic records to one of a plurality of worker computing devices selected based on the determination and on at least one of the first hash and the second hash;
transmitting, by the distribution server, to each of the plurality of worker computers in the computer network, a request for an enumeration of Internet Protocol (IP) addresses ranked according to a criterion;
receiving, by the distribution server, from the first of the plurality of worker computers, a first partial enumeration of the requested IP addresses ranked according to the criterion, the first partial enumeration stored in a hash table;
receiving, by the distribution server, from a second of the plurality of worker computers, a second partial enumeration of the requested IP addresses ranked according to the criterion, the second partial enumeration stored in a hash table;
generating, by the distribution server, a combined enumeration including the first partial enumeration and the second partial enumeration, the combined enumeration ranked according to the criterion;
deduplicating, by the distribution server, the combined enumeration; and
providing, by the distribution server, a network security assessment based on the deduplicated combined enumeration.
2. The method of claim 1 , wherein receiving further comprises receiving, by the distribution server, from the exporter device, the plurality of network traffic records, each of the plurality of network traffic records including a source Internet Protocol (IP) address.
3. The method of claim 1 , wherein receiving further comprises receiving, by the distribution server, from the exporter device, the plurality of network traffic records, each of the plurality of network traffic records including a destination Internet Protocol (IP) address.
4. The method of claim 1 , wherein receiving further comprises receiving, by the distribution server, from the exporter device, the plurality of network traffic records, each of the plurality of network traffic records identifying a communications protocol.
5. The method of claim 1 , wherein receiving further comprises receiving, by the distribution server, from the exporter device, the plurality of network traffic records, each of the plurality of network traffic records identifying at least one communications port.
6. The method of claim 1 , wherein generating the first hash further comprises executing a hashing function using a numerical value associated with a source Internet Protocol (IP) address and a numerical value associated with a destination Internet Protocol (IP) address.
7. The method of claim 1 , wherein generating the second hash further comprises executing a hashing function using a numerical value associated with a source Internet Protocol (IP) address and a numerical value associated with a destination Internet Protocol (IP) address.
8. The method of claim 1 , wherein transmitting the request for the enumeration of IP addresses further comprises transmitting a request for an enumeration of IP addresses identified as a destination address in each of a plurality of network traffic records, the enumeration ranked according to the criterion.
9. The method of claim 1 , wherein transmitting the request for the enumeration of IP addresses further comprises transmitting a request for an enumeration of IP addresses identified as a source address in each of a plurality of network traffic records, the enumeration ranked according to the criterion.
10. The method of claim 1 , wherein transmitting the request for the enumeration of IP addresses further comprises transmitting a request for an enumeration of IP addresses associated with a protocol identified in each of a plurality of network traffic records, the enumeration ranked according to the criterion.
11. The method of claim 1 , wherein transmitting the request for the enumeration of IP addresses further comprises transmitting a request for an enumeration of IP addresses associated with a port identified in each of a plurality of network traffic records, the enumeration ranked according to the criterion.
12. The method of claim 1 , wherein receiving, by the distribution server, from the first of the plurality of worker computers, the first partial enumeration of the requested IP addresses further comprises:
receiving, by the distribution server, a result of a search, by the first of the plurality of worker computers, for at least one IP address satisfying the criterion and associated with at least one of the first of the plurality of network traffic records and the second of the plurality of traffic records transmitted to the first of the plurality of worker computing devices by the distribution server.
13. The method of claim 1 , wherein receiving, by the distribution server, from the second of the plurality of worker computers, the second partial enumeration of the requested IP addresses further comprises:
receiving, by the distribution server, a result of a search, by the second of the plurality of worker computers, for at least one IP address satisfying the criterion and associated with at least one of the first of the plurality of network traffic records and the second of the plurality of traffic records transmitted to the second of the plurality of worker computing devices by the distribution server.
14. A non-transitory computer readable medium comprising computer program instructions tangibly stored on the computer readable medium, wherein the computer program instructions are executable by at least one computer processor to perform a method for generating a combined, deduplicated enumeration of network traffic records received from a plurality of worker computers and providing a network security assessment based on the enumeration, the method comprising:
receiving, by a distribution server in a computer network, from an exporter device, a plurality of network traffic records;
generating, by the distribution server, a first hash from a first plurality of fields in a first of the plurality of network traffic records;
generating, by the distribution server, a second hash from a second plurality of fields in a second of the plurality of network traffic records;
comparing, by the distribution server, the first hash and the second hash;
determining, by the distribution server, that the first of the plurality of network traffic records and the second of the plurality of traffic records relate to a session, based upon the comparison;
transmitting, by the distributions server, the first of the plurality of network traffic records and the second of the plurality of traffic records to one of a plurality of worker computing devices selected based on the determination and on at least one of the first hash and the second hash;
transmitting, by the distribution server, to each of the plurality of worker computers in the computer network, a request for an enumeration of Internet Protocol (IP) addresses ranked according to a criterion;
receiving, by the distribution server, from the first of the plurality of worker computers, a first partial enumeration of the requested IP addresses ranked according to the criterion, the first partial enumeration stored in a hash table;
receiving, by the distribution server, from a second of the plurality of worker computers, a second partial enumeration of the requested IP addresses ranked according to the criterion, the second partial enumeration stored in a hash table;
generating, by the distribution server, a combined enumeration including the first partial enumeration and the second partial enumeration, the combined enumeration ranked according to the criterion;
deduplicating, by the distribution server, the combined enumeration; and
providing, by the distribution server, a network security assessment based on the deduplicated combined enumeration.