Iterative development of protocol parsers
Systems, methods, and related technologies for determining fields of an unknown protocol are described. One or more packets may be removed from a network traffic capture in response to the one or more packets having a known protocol. The remaining network traffic capture may be grouped into one or more clusters of packets based on similarity. Each of the one or more clusters may be parsed to identify one or more fields of an unknown protocol. The network traffic capture may be modified, including annotating the one or more fields of the unknown protocol.
1. A method, comprising:
removing one or more packets from a network traffic capture in response to the one or more packets having a known protocol;
grouping the network traffic capture without the one or more packets having the known protocol, into one or more clusters of packets based on similarity;
parsing each of the one or more clusters to identify one or more fields of an unknown protocol;
modifying the network traffic capture including annotating the identified one or more fields of the unknown protocol; and
generating without user input, a protocol parser, including parsing each of the annotated one or more fields of the unknown protocol to generate a description of the unknown protocol comprising identified one or more fields of the unknown protocol and an order of the identified one or more fields of the unknown protocol, and compiling the description into the protocol parser.
2. The method of claim 1 , wherein the network traffic is grouped based on a common host type.
3. The method of claim 1 , wherein the network traffic is grouped based on a common frequency of communication or time of communication.
4. The method of claim 1 , wherein the network traffic is grouped based on a common port number.
5. The method of claim 1 , wherein the network traffic is grouped based on a common network protocol layer.
6. The method of claim 1 , wherein the network traffic is grouped based on a common packet length.
7. The method of claim 1 , wherein parsing each of the one or more clusters to identify the one or more fields of the unknown protocol includes identifying a string, an integer, a timestamp, or a constant repeating value.
8. The method of claim 7 , wherein parsing each of the one or more clusters to identify the one or more fields of the unknown protocol includes calling an application programming interface (API) of a field modeling tool to identify the string, the integer, the timestamp, or the repeating value.
9. The method of claim 1 , wherein obtaining the network traffic capture includes presenting a web-based application that receives and aggregates network traffic from one or more users.
10. The method of claim 1 , further comprising storing the one or more fields of the unknown protocol in computer-readable memory, and repeating the method to modify the network traffic capture including annotating additional one or more fields of the unknown protocol.
11. The method of claim 10 , further comprising presenting an alert in response to the additional one or more fields of the unknown protocol being identified.
12. A system, comprising:
a memory; and
a processing device, operatively coupled to the memory, to:
remove one or more packets from a network traffic capture in response to the one or more packets having a known protocol;
group the network traffic capture without the on or more packets having the known protocol, into one or more clusters of packets based on similarity;
parse each of the one or more clusters to identify one or more fields of an unknown protocol;
modify the network traffic capture including annotating the identified one or more fields of the unknown protocol; and
generating without user input, a protocol parser, including parsing each of the annotated one or more fields of the unknown protocol to generate a description of the unknown protocol comprising identified one or more fields of the unknown protocol and an order of the identified one or more fields of the unknown protocol, and compiling the description into the protocol parser.
13. The system of claim 12 , wherein the network traffic is grouped based on a common host type.
14. The system of claim 12 , wherein the network traffic is grouped based on a common frequency of communication or time of communication.
15. The system of claim 12 , wherein the network traffic is grouped based on a common port number.
16. The system of claim 12 , wherein the network traffic is grouped based on a common network protocol layer.
17. The system of claim 12 , wherein the network traffic is grouped based on a common packet length.
18. A non-transitory computer readable medium having instructions encoded thereon that, when executed by a processing device, cause the processing device to:
remove one or more packets from a network traffic capture in response to the one or more packets having a known protocol;
group the network traffic capture without the one or more packets having the known protocol, into one or more clusters of packets based on similarity;
parse each of the one or more clusters to identify one or more fields of an unknown protocol;
modify the network traffic capture including annotating the identified one or more fields of the unknown protocol; and
generating without user input, a protocol parser, including parsing each of the annotated one or more fields of the unknown protocol to generate a description of the unknown protocol comprising identified one or more fields of the unknown protocol and an order of the identified one or more fields of the unknown protocol, and compiling the description into the protocol parser.
19. The non-transitory computer readable medium of claim 18 , wherein the network traffic is grouped based on a common host type.