System and method for automated generation of web decoding templates
Methods and systems for decoding communication protocols having an unknown structure. In the disclosed embodiments, a decoding system analyzes network traffic that uses such a communication protocol, and semi-automatically generates a structured template for decoding the protocol. In an example embodiment, the traffic comprises HTTP transactions used in some unknown variant of a Web-based e-mail or social network application, and the system generates an Extensible Markup Language (XML) template for parsing such transactions. The system enables an analyst to review sample transactions, and identify target components of the protocol that contain target information of interest. The system typically generates a set of rules with the assistance of the analyst.
1. A method, comprising:
receiving, via a network interface of a decoding system, sample network packets exchanged with a web server that correspond with a target transaction type, wherein the packets are exchanged using a transfer protocol, wherein the packets comprise data of a target communication protocol with an unknown structure;
presenting, via a graphical user interface run by a decoding processor of the monitoring system, one or more of the packets on a display of an operator terminal;
receiving, by the decoding processor, one or more inputs from the operator terminal for formulating one or more recognition rules for recognizing target fields in the data of the target communication protocol, and one or more extraction rules for extracting target information from the recognized target fields;
generating, by the decoding processor, a structured template for parsing the target transaction type of the target communication protocol based on the one or more recognition rules and the one or more extraction rules;
receiving, via the network interface, subsequent network packets exchanged between one or more user computers and the web server, the subsequent network packets are encoded with the target communication protocol; and
decoding, by the decoding processor, the subsequent network packets using the one or more recognition rules and the one or more extraction rules of the structured template to extract the target information from the subsequent network packets.
2. The method according to claim 1 , and comprising formulating one or more producer rules that specify actions to be applied to the target information, wherein extracting the target information comprises acting on the extracted target information in accordance with the producer rules.
3. The method according to claim 1 , wherein formulating the recognition rules and the extraction rules comprises identifying two or more occurrences of a target component or a target information item in the network traffic, and defining the rules so as to match the two or more occurrences.
4. The method according to claim 3 , wherein defining the rules comprises automatically generating a regular expression that matches the two or more occurrences.
5. The method according to claim 1 , wherein formulating the recognition rules and the extraction rules comprises testing one or more of the rules by applying the one or more of the rules to sample network traffic.
6. The method according to claim 1 , further comprising:
generating, by the decoding processor, a second structured template for parsing a second transaction type of the target communication protocol based on one or more second recognition rules and one or more second extraction rules, wherein a common target field present in both the target transaction type and the second transaction type is represented only once in the structured template and the second structured template.
7. The method according to claim 1 , wherein the web server hosts a Web-based application, and wherein the target communication protocol pertains to the Web-based application.
8. The method according to claim 7 , wherein the Web-based application comprises one of a Web-based e-mail application, an instant-messaging application and a social network application.
9. Apparatus, comprising:
a network interface configured to receive sample network packets exchanged with a web server that correspond with a target transaction type, wherein the packets are exchanged using a transfer protocol, and wherein the packets comprise data of a target communication protocol with an unknown structure,
wherein the network interface is further configured to receive subsequent network packets exchanged between one or more user computers and the web server, the subsequent network packets are encoded with the target communication protocol;
a non-transitory memory, which is configured to store the sample network traffic; and
a processor computer programmed in software, which, when the software is executed on the processor computer, is configured to:
present, via a graphical user interface, one or more of the packets on a display of an operator terminal,
receive one or more inputs from the operator terminal to formulate one or more recognition rules for recognizing target fields in the data of the target communication protocol and one or more extraction rules for extracting target information from the recognized target fields,
generate a structured template for parsing the target transaction type of the target communication protocol based on the one or more recognition rules and the one or more extraction rules, and
decode the subsequent network packets using the one or more recognition rules and the one or more extraction rules of the structured template to extract the target information from the subsequent network packets.
10. The apparatus according to claim 9 , wherein the processor is configured to formulate one or more producer rules that specify actions to be applied to the target information, and to act on the extracted target information in accordance with the producer rules.
11. The apparatus according to claim 9 , wherein the processor is configured to identify two or more occurrences of a target component or a target information item in the network traffic, and to define the recognition rules or the extraction rules so as to match the two or more occurrences.
12. The apparatus according to claim 11 , wherein the processor is configured to automatically generate a regular expression that matches the two or more occurrences.
13. The apparatus according to claim 9 , wherein the processor is configured to test one or more of the recognition rules or the extraction rules by applying the one or more of the rules to sample network traffic.
14. The apparatus according to claim 9 , wherein the processor is further configured to:
generate a second structured template for parsing a second transaction type of the target communication protocol based on one or more second recognition rules and one or more second extraction rules, wherein a common target field present in both the target transaction type and the second transaction type is represented only once in the structured template and the second structured template.
15. The apparatus according to claim 9 , wherein the web server hosts a Web-based application, and wherein the target communication protocol pertains to the Web-based application.
16. The apparatus according to claim 15 , wherein the Web-based application comprises one of a Web-based e-mail application, an instant-messaging application and a social network application.
17. The method according to claim 1 , wherein the transfer protocol is the Hyper-Text Transfer Protocol (HTTP) and the sample network packets are HTTP request and response packets.
18. The method according to claim 1 , wherein the sample network packets are exchanged between the decoding system and the web server, wherein the sample network packets include known target information in one or more of the target fields.
19. The apparatus according to claim 9 , wherein the transfer protocol is the Hyper-Text Transfer Protocol (HTTP) and the sample network packets are HTTP request and response packets.
20. The apparatus according to claim 9 , wherein the sample network packets are exchanged between the apparatus and the web server, wherein the sample network packets include known target information in one or more of the target fields.