SYSTEM AND METHOD FOR CORRELATING CAPTURED NETWORK PACKET WITH METADATA
A new approach is proposed that contemplates system and method to support correlating captured network traffic going through a network appliance with its metadata for troubleshooting and debugging of the network traffic. First, network traffic in the form of data packets are captured and filtered for metadata based on user-specified parameters. A correlation is then established between the captured data packets and the metadata in a chronologically correct fashion. The metadata is then integrated and stored with the captured network packets in a correlated network traffic database in a standard storage format, which can later be retrieved for visualization, debugging and tracing of program flows and other data with respect to the network traffic through the network appliance.
1 . A system to support correlating captured network packet with metadata, comprising:
a metadata correlating engine running on a host, which in operation, is configured to
compile and set a plurality of user-specified parameters for capturing one or more network packets and related metadata in network traffic from a network appliance;
capture the network packets and the related metadata when network traffic capturing is triggered based on the plurality of set parameters;
correlate and interleave the network packets with the related metadata in chronological order;
store the metadata-correlated network packets in a standard storage format in a correlated network traffic database for further visualization, analysis, and debugging of the network traffic from the network appliance by the user;
said correlated network traffic database running on the host, which in operation, is configured to maintain the metadata-correlated network packets in a standard storage format for further visualization, analysis, and debugging.
2 . The system of claim 1 , wherein:
the network appliance is a hardware-based, software-programmable networking device that includes a firewall configurable by software at runtime.
3 . The system of claim 1 , wherein:
the metadata includes runtime program flow and tracing information of the network packets.
4 . The system of claim 1 , wherein:
the standard storage format is packet capture (PCAP) or PCAP Next Generation Dump File Format (PCAP-NG).
5 . The system of claim 1 , wherein:
the metadata correlating engine is configured to store the metadata with the network packets outside of annotation fields without requiring any modification to the standard storage format.
6 . The system of claim 1 , wherein:
the metadata correlating engine comprises
an network packet handler running in kernel space of operating system (OS) of the host configured to capture and forward the network packets and the metadata; and
a control application running in user space of the OS of the host configured to control the network packet handler and correlate and save the metadata-correlated network packets to the correlated network traffic database.
7 . The system of claim 6 , wherein:
the network packet handler is configured to transfer a copy of the captured network packets and the metadata to a special local communication socket in the user space, wherein the local communication socket is a minimal overhead communication interface between the network packet handler in the kernel space and the control application in the user space.
8 . The system of claim 6 , wherein:
the network packet handler is configured to transfer a copy of the captured network packets and the metadata to a remote UDP socket to a service.
9 . The system of claim 6 , wherein:
the control application is configured to compile and set the user-specified parameters for packet capturing and metadata filtering into an structure that is used by the network packet handler in the kernel space.
10 . The system of claim 7 , wherein:
the control application is configured to listen to the special local communication socket for the incoming network packets and/or metadata from the network appliance as captured based on the user-specified parameters.
11 . The system of claim 7 , wherein:
the control application is configured to inject additional metadata into the captured network packet via the local communication socket.
12 . The system of claim 7 , wherein:
the network packet handler is configured to copy and wrap the network packets with a header and transfers it to the local communication socket if one or more matching network packets are found and a callback is triggered.
13 . The system of claim 12 , wherein:
the header is a Layer 2 pseudo protocol header required to differentiate and visualize the metadata wrapped in the network packets by the user via network analyzing tools with a protocol dissector script added as a plugin.
14 . The system of claim 12 , wherein:
the special local communication socket is configured to adopt a specific communication protocol for communication between the network packet handler and the control application, including setting the parameters of the network packet handler and transferring the wrapped network packets and metadata back to the control application.
15 . The system of claim 1 , wherein:
the user-specified parameters limit network traffic to between specified source IPs and ports and specified destination IPs and ports including complete network IP and/or port ranges.
16 . The system of claim 1 , wherein:
the user-specified parameters include one or more corresponding session slot filter settings when a current session is created, wherein session slot filter settings are matched with the network packets to identify the metadata for metadata filtering.
17 . A method to support correlating captured network packet with metadata, comprising:
compiling and setting a plurality of user-specified parameters for capturing one or more network packets and related metadata in network traffic from a network appliance;
capturing the network packets and the related metadata when network traffic capturing is triggered based on the plurality of set parameters;
correlating and interleaving the network packets with the related metadata in chronological order;
storing the metadata-correlated network packets in a standard storage format in a correlated network traffic database for further visualization, analysis, and debugging of the network traffic from the network appliance by the user.
18 . The method of claim 17 , further comprising:
storing the metadata with the network packets outside of annotation fields without requiring any modification to the standard storage format.
19 . The method of claim 17 , further comprising:
capturing and forwarding the network packets and the metadata via an network packet handler running in kernel space of operating system (OS) of the host;
controlling the network packet handler and correlating and saving the metadata-correlated network packets to the correlated network traffic database via a control application running in user space of the OS of the host.
20 . The method of claim 19 , further comprising:
transferring a copy of the captured network packets and the metadata to a special local communication socket in the user space, wherein the local communication socket is a minimal overhead communication interface between the network packet handler in the kernel space and the control application in the user space.
21 . The method of claim 19 , further comprising:
transferring a copy of the captured network packets and the metadata to a remote UDP socket to a service.
22 . The method of claim 19 , further comprising:
compiling and setting the user-specified parameters for packet capturing and metadata filtering into an structure that is used by the network packet handler in the kernel space.
23 . The method of claim 20 , further comprising:
listening to the special local communication socket for the incoming network packets and/or metadata from the network appliance as captured based on the user-specified parameters.
24 . The method of claim 20 , further comprising:
injecting additional metadata into the captured network packet via the local communication socket.
25 . The method of claim 20 , further comprising:
copying and wrapping the network packets with a header and transfers it to the local communication socket if one or more matching network packets are found and a callback is triggered, wherein the header is a Layer 2 pseudo protocol header.
26 . The method of claim 25 , further comprising:
differentiating and visualizing the metadata wrapped in the network packets by the user via network analyzing tools with a protocol dissector script added as a plugin.
27 . The method of claim 25 , further comprising:
adopting a specific communication protocol for communication between the network packet handler and the control application, including setting the parameters of the network packet handler and transferring the wrapped network packets and metadata back to the control application.