Iterative development of protocol parsers
Systems, methods, and related technologies for determining fields of an unknown protocol are described. Network traffic capture is grouped into one or more clusters of packets based on similarity. Each of the one or more clusters are parsed to identify one or more fields of an unknown protocol. The network traffic capture is modified, including annotating the identified one or more fields of the unknown protocol. A protocol parser is generated without user input, including parsing each of the annotated one or more fields of the unknown protocol to generate a description of the unknown protocol comprising identified one or more fields of the unknown protocol and an order of the identified one or more fields of the unknown protocol, and compiling the description into the protocol parser.
1. A method, comprising:
grouping network traffic capture into one or more clusters of packets based on similarity;
parsing each of the one or more clusters to identify one or more fields of an unknown protocol;
modifying the network traffic capture including annotating the identified one or more fields of the unknown protocol; and
generating, by a processing device, without user input, a protocol parser, including parsing each of the annotated one or more fields of the unknown protocol to generate a description of the unknown protocol comprising identified one or more fields of the unknown protocol and an order of the identified one or more fields of the unknown protocol, and compiling the description into the protocol parser.
2. The method of claim 1 , wherein the network traffic capture is grouped based on commonality of one or more of: a host type, a frequency of communication, a time of communication, a port number, a network protocol layer, or a packet length.
3. The method of claim 1 , wherein annotating the identified one or more fields is performed with a visual indicator.
4. The method of claim 1 , wherein parsing each of the one or more clusters includes detecting a pattern of a repeating value.
5. The method of claim 4 , wherein the repeating value is associated with a constant value.
6. The method of claim 1 , wherein parsing each of the one or more clusters includes detecting a signature of a field type.
7. The method of claim 6 , wherein the signature of the field type is associated with a string, an integer, or a timestamp.
8. The method of claim 1 , wherein the network traffic capture is obtained using a web crawler.
9. The method of claim 1 , wherein the method is performed in a first iteration to generate part of the protocol parser, and the method is repeated one or more second iterations to identify additional fields of the unknown protocol until the protocol parser satisfies a threshold.
10. The method of claim 9 , wherein the threshold includes one or more of a number of the one or more fields that are identified, or a percentage of the one or more fields that are identified.
11. A system, comprising:
a memory; and
a processing device, operatively coupled to the memory, to:
group network traffic capture into one or more clusters of packets based on similarity;
parse each of the one or more clusters to identify one or more fields of an unknown protocol;
modify the network traffic capture including to annotate the identified one or more fields of the unknown protocol; and
generate, by the processing device, a protocol parser without user input, including to parse each of the annotated one or more fields of the unknown protocol to generate a description of the unknown protocol that comprises the identified one or more fields of the unknown protocol and an order of the identified one or more fields of the unknown protocol, and to compile the description into the protocol parser.
12. The system of claim 11 , wherein the network traffic capture is grouped based on commonality of one or more of: a host type, a frequency of communication, a time of communication, a port number, a network protocol layer, or a packet length.
13. The system of claim 11 , wherein to annotate the identified one or more fields is performed with a visual indicator.
14. The system of claim 11 , wherein to parse each of the one or more clusters includes detecting a pattern of a repeating value.
15. The system of claim 14 , wherein the repeating value is associated with a constant value.
16. A non-transitory computer readable medium having instructions encoded thereon that, when executed by a processing device, cause the processing device to:
group network traffic capture into one or more clusters of packets based on similarity;
parse each of the one or more clusters to identify one or more fields of an unknown protocol;
modify the network traffic capture including to annotate the identified one or more fields of the unknown protocol; and
generate, by the processing device, a protocol parser without user input, including: to parse each of the annotated one or more fields of the unknown protocol to generate a description of the unknown protocol that comprises the identified one or more fields of the unknown protocol and an order of the identified one or more fields of the unknown protocol, and to compile the description into the protocol parser.
17. The non-transitory computer readable medium of claim 16 , wherein to parse each of the one or more clusters includes to detect a signature of a field type.
18. The non-transitory computer readable medium of claim 17 , wherein the signature of the field type is associated with a string, an integer, or a timestamp.
19. The non-transitory computer readable medium of claim 16 , wherein the network traffic capture is obtained using a web crawler.
20. The non-transitory computer readable medium of claim 16 , wherein the instructions are performed in a first iteration to generate part of the protocol parser, and the instructions are repeated one or more second iterations to identify additional fields of the unknown protocol until the protocol parser satisfies a threshold.