Network flow data ingestion, storage, and analysis
The Kentik Data Engine (KDE) is an integrated real-time, big data software system able to analyze what exactly is happening on a network at the present moment, and what happened on the network over the a prior period of time. KDE collects live operational data from computer network infrastructure devices (routers and switches) and computer hosts, consisting of multiple data types, categories, and protocols, and correlates them to analyze network activity and health. KDE does this in a lossless manner, meaning that it retains all raw data rather than summarizing or aggregating prior to storage. In this way, KDE provides a combination of precise, actionable information in real-time as well as a complete forensic data store for detailed exploratory analysis.
1. A method for network monitoring, comprising:
receiving network flow data represented as a plurality of data records describing operational data for a device in a computer network;
ingesting the received network flow data for storage within a plurality of data servers, by:
identifying a subset of the plurality of data servers for storing data records for the device, based on a combination of the device and a time period during which the network flow data was ingested;
forwarding the data records for the device to the identified subset of the plurality of data servers;
recording metadata describing which data servers of the plurality of data servers are in the subset storing the data records for the device, wherein each data server of the subset of data servers is associated with metadata specifying at least one device and ingestion time period combination;
responsive to receiving a query for at least a portion of the data records for the device:
dividing the received query into a plurality of subqueries, each corresponding to a different device and ingestion time period combination associated with the received query;
identifying which data servers of the plurality of data servers are in the subset storing the data records for the device, by identifying for each subquery, using the recorded metadata, at least one data server of the plurality of data servers storing data records corresponding to the device and ingestion time period combination specified by the subquery;
transmitting the query to at least one of the identified data servers storing the data records for the device; and
receiving the data records for the device from the at least one of the identified data servers.
2. The method of claim 1 , wherein receiving network flow data comprises:
receiving the network flow data from the device via the computer network;
validating a sending address from which the network flow data was received;
responsive to validation of the sending address, enriching the received network flow data with enrichment data describing the device; and
forming the plurality of data records using the enriched network flow data.
3. The method of claim 1 , wherein receiving network flow data comprises:
receiving the plurality of data records from a software agent executing on a machine on a local area network associated with the device; and
validating the device to affirm that the agent is eligible to receive network flow data for the device;
wherein the plurality of data records received from the software agent are forwarded to the identified subset of the plurality of data servers responsive to validation of the device.
4. The method of claim 1 , wherein each data record comprises a data tuple identifying traffic between two hosts on the computer network at an application level.
5. The method of claim 1 , wherein each data record for the device is stored on at least two different data servers of the plurality of data servers, and wherein the method further comprises:
determining a load level of each data server of the identified data servers, based upon a number of outstanding queries associated with each data server; and
transmitting the query to receive the data record to the identified data server having a lowest load level.
6. The method of claim 1 , further comprising:
forwarding the data records for the device to an alert subsystem;
at the alert subsystem, aggregating a subset of the data records corresponding to a particular time interval;
comparing the aggregated subset of data records with one or more trigger rules, wherein an alert state is triggered in response to an attribute of the aggregated subset of data records exceeding a value specified by the one or more trigger rules; and
taking a notification action responsive to the alert state being triggered.
7. A system for network monitoring, comprising:
a plurality of ingest servers configured to:
receive network flow data represented as a plurality of data records describing operational data for a device in a computer network;
identify a subset of a plurality of data servers for storing data records for the device, based on a combination of the device and a time period during which the network flow data was ingested;
forward the data records for the device to the identified subset of the plurality of data servers;
a metadata system configured to record metadata describing which data servers of the plurality of data servers are in the subset storing the data records for the device, wherein each data server of the subset of data servers is associated with metadata specifying at least one device and ingestion time period combination;
a query processor configured to:
receive a query for at least a portion of the data records for the device;
divide the received query into a plurality of subqueries, each corresponding to a different device and ingestion time period combination associated with the received query;
identify which data servers of the plurality of data servers are in the subset storing the data records for the device, by identifying for each subquery, using the recorded metadata, at least one data server of the plurality of data servers storing data records corresponding to the device and ingestion time period combination specified by the subquery;
transmit the query to at least one of the identified data servers that are storing the data records for the device; and
receive the data records for the device from the at least one of the identified data servers.
8. The system of claim 7 , wherein plurality of ingest servers are further configured to:
receive the network flow data from the device via the computer network;
validate a sending address from which the network flow data was received;
responsive to validation of the sending address, enrich the received network flow data with enrichment data describing the device; and
form the plurality of data records using the enriched network flow data.
9. The system of claim 7 , wherein plurality of ingest servers are further configured to:
receive the plurality of data records from a software agent executing on a machine on a local area network associated with the device; and
validate the device to affirm that the agent is eligible to receive network flow data for the device;
wherein the plurality of data records received from the software agent are forwarded to the identified subset of the plurality of data servers responsive to validation of the device.
10. The system of claim 7 , wherein each data record for the device is stored on at least two different data servers of the plurality of data servers, and the query processor is further configured to:
determine a load level of each data server of the identified data servers, based upon a number of outstanding queries associated with each data server; and
transmit the query to receive the data record to the identified data server having a lowest load level.
11. The system of claim 7 , wherein the plurality of ingest servers are further configured to forward the data records for the device to an alert subsystem, the alert subsystem configured to:
aggregate a subset of the data records corresponding to a particular time interval;
compare the aggregated subset of data records with one or more trigger rules, wherein an alert state is triggered in response to an attribute of the aggregated subset of data records exceeding a value specified by the one or more trigger rules; and
take a notification action responsive to the alert state being triggered.
12. A computer program product for performing network monitoring, the computer program product comprising a non-transitory computer-readable storage medium containing computer program code that when executed causes one or more processors to perform:
receiving network flow data represented as a plurality of data records describing operational data for a device in a computer network;
ingesting the received network flow data for storage within a plurality of data servers, by:
identifying a subset of the plurality of data servers for storing data records for the device, based on a combination of the device and a time period during which the network flow data was ingested;
forwarding the data records for the device to the identified subset of the plurality of data servers;
recording metadata describing which data servers of the plurality of data servers are in the subset storing the data records for the device, wherein each data server of the subset of data servers is associated with metadata specifying at least one device and ingestion time period combination;
responsive to receiving a query for at least a portion of the data records for the device:
dividing the received query into a plurality of subqueries, each corresponding to a different device and ingestion time period combination associated with the received query;
identifying which data servers of the plurality of data servers are in the subset storing the data records for the device, by identifying for each subquery, using the recorded metadata, at least one data server of the plurality of data servers storing data records corresponding to the device and ingestion time period combination specified by the subquery;
transmitting the query to at least one of the identified data servers storing the data records for the device; and
receiving the data records for the device from the at least one of the identified data servers.
13. The computer program product of claim 12 , wherein the computer program code further causes the one or more processors to perform:
receiving the network flow data from the device via the computer network;
validating a sending address from which the network flow data was received;
responsive to validation of the sending address, enriching the received network flow data with enrichment data describing the device; and
forming the plurality of data records using the enriched network flow data.
14. The computer program product of claim 12 , wherein the computer program code further causes the one or more processors to perform:
forwarding the data records for the device to an alert subsystem;
at the alert subsystem, aggregating a subset of the data records corresponding to a particular time interval;
comparing the aggregated subset of data records with one or more trigger rules, wherein an alert state is triggered in response to an attribute of the aggregated subset of data records exceeding a value specified by the one or more trigger rules; and
taking a notification action responsive to the alert state being triggered.
15. The computer program product of claim 12 , wherein each data record for the device is stored on at least two different data servers of the plurality of data servers, and wherein the method further comprises:
determining a load level of each data server of the identified data servers, based upon a number of outstanding queries associated with each data server; and
transmitting the query to receive the data record to the identified data server having a lowest load level.