Systems and methods for hierarchical deep packet inspection for scalable network monitoring and cyber security functions
A system and method for hierarchical network monitoring functions are disclosed. An order of execution for layer functions of a network architecture is determined. The layer functions may be distributed across multiple layers. The layers may include a sensor layer, a federated application layer, and a data lake layer. A machine learning model may be executed at a first layer. The first layer may be the sensor layer.
1 . A system, comprising:
a data processing system comprising one or more processors coupled with memory, the data processing system configured to:
collect a plurality of network data packets from user equipment, network equipment, or monitoring equipment connected to a communications network;
determine an order for executing a plurality of layer functions of a hierarchical network architecture based on a priority for each layer function, wherein the hierarchical network architecture comprises a first layer, a second layer, and a third layer, the first layer being a sensor layer, the second layer being a federated application layer, and the third layer being a data lake layer, and wherein the number of layer functions in the first layer is greater than the number of layer functions in the second layer, which is greater than the number of layer functions in the third layer;
execute the plurality of layer functions according to the order using the plurality of network data packets and historical network data as input to generate network data associated with the communications network; and
execute at least one machine learning model in the sensor layer using deep packet inspection data to adaptively filter traffic and detect security threats;
adjust the communications network according to the generated network data.
2 . The system of claim 1 , wherein:
the first quantity of layer functions at the first layer comprises a traffic analysis layer function, a cloud threat detection layer function, an IDP engine layer function, a knowledge base framework layer function, and a threat intelligence feed layer function;
the second quantity of layer functions at the second layer comprises a threat detection layer function, a retrospective analysis layer function, an application mapping layer function, and an asset validation layer function; and
the third quantity of layer functions at the third layer comprises an automated response layer function, a telemetry data layer function, and a threat intelligence correlation layer function.
3 . The system of claim 1 , wherein the network architecture is a hierarchical adaptive service intelligence (ASI) architecture, a hierarchical deep packet inspection (DPI) architecture, or a combination thereof, wherein the ASI architecture comprises four layers.
4 . The system of claim 1 , wherein the order for executing the plurality of layer functions comprises instructions to:
first, executing the first quantity of layer functions at the first layer;
second, executing the second quantity of layer functions at the second layer; and
third, executing the third quantity of layer functions at the third layer.
5 . The system of claim 1 , wherein the data processing system is further configured to:
execute each layer function of each quantity of layer functions at each layer separately from other layer functions at each layer.
6 . A method, comprising:
collecting, by one or more processors, a plurality of network data packets from user equipment, network equipment, or monitoring equipment connected to a communications network;
determining, by the one or more processors, an order for executing a plurality of layer functions of a hierarchical network architecture based on a priority for each layer function, wherein the hierarchical network architecture comprises a first layer, a second layer, and a third layer, the first layer being a sensor layer, the second layer being a federated application layer, and the third layer being a data lake layer, and wherein the number of layer functions in the first layer is greater than the number of layer functions in the second layer, which is greater than the number of layer functions in the third layer;
executing, by the one or more processors, the plurality of layer functions according to the order using the plurality of network data packets and historical network data as input to generate network data associated with the communications network; and
executing, by the one or more processors, at least one machine learning model in the sensor layer using deep packet inspection data to adaptively filter traffic and detect security threats;
adjusting, by the one or more processors, the communications network according to the generated network data.
7 . The method of claim 6 , wherein:
the first quantity of layer functions at the first layer comprises a traffic analysis layer function, a cloud threat detection layer function, an IDP engine layer function, a knowledge base framework layer function, and a threat intelligence feed layer function;
the second quantity of layer functions at the second layer comprises a threat detection layer function, a retrospective analysis layer function, an application mapping layer function, and an asset validation layer function; and
the third quantity of layer functions at the third layer comprises an automated response layer function, a telemetry data layer function, and a threat intelligence correlation layer function.
8 . The method of claim 6 , wherein the network architecture is a hierarchical adaptive service intelligence (ASI) architecture, a hierarchical deep packet inspection (DPI) architecture, or a combination thereof, wherein the ASI architecture comprises four layers.
9 . The system of claim 1 , wherein the order for executing the plurality of layer functions comprises instructions to:
first, executing the first quantity of layer functions at the first layer;
second, executing the second quantity of layer functions at the second layer; and
third, executing the third quantity of layer functions at the third layer.
10 . The method of claim 6 , further comprising:
executing, by the one or more processors, each layer function of each quantity of layer functions at each layer separately from other layer functions at each layer.
11 . The system of claim 1 , wherein the data processing system is further configured to:
execute the plurality of layer functions at the first layer to filter noise from the plurality of network data packets before executing the plurality of layer functions at the second layer, thereby reducing a quantity of data processed at the second layer and the third layer.
12 . The system of claim 1 , wherein the hierarchical network architecture is configured to reduce mean time to know (MTTK) for detecting security threats by executing the plurality of layer functions at the first layer in real-time and executing the plurality of layer functions at the second layer and the third layer on aggregated data from the first layer.
13 . The system of claim 1 , wherein:
the first layer comprises a traffic analysis layer function that analyzes packet headers and payload data to identify traffic patterns;
the second layer comprises a threat detection layer function that correlates the traffic patterns identified by the first layer with known threat signatures; and
the third layer comprises a threat intelligence correlation layer function that correlates detected threats with external threat intelligence feeds.
14 . The system of claim 1 , wherein the data processing system is further configured to:
store metadata and packet decodes from the plurality of network data packets at the first layer for retrospective analysis by the second layer.
15 . The system of claim 1 , wherein the at least one machine learning model comprises an ASI Flow machine learning model that conducts network behavioral analysis to generate deterministic signatures that reduce false positives.
16 . The system of claim 1 , wherein the data processing system is further configured to:
convert raw network data packets into layer 2 - 7 metadata using Deep Packet Inspection and Adaptive Service Intelligence (ASI) at the first layer; and
provide the layer 2 - 7 metadata to the second layer and the third layer for analysis.
17 . The system of claim 1 , wherein the data processing system is further configured to:
match indicators of compromise (IoCs) from a threat intelligence feed at the first layer;
match Suricata-based rules and signatures at the first layer; and
when a threat is detected at the first layer, map the threat to known MITRE ATT&CK tactics, techniques, and procedures (TTPs) for analysis at the second layer.
18 . The system of claim 1 , wherein the hierarchical network architecture is an Omnis network security architecture comprising:
network instrumentation at the first layer that performs deep packet inspection to convert raw packets into metadata;
network instrumentation at the second layer that performs federated application analysis; and
a data lake at the third layer that stores and analyzes aggregated network data.
19 . The system of claim 1 , wherein the data processing system is further configured to:
conduct forensic investigation at the second layer using the historical network data stored at the first layer to validate threats detected in real-time and reduce mean time to repair (MTTR).