Automatic parsing of text-based application protocols using network traffic data

1. A method for analyzing a protocol of a network, comprising:

obtaining a plurality of conversations from the network, wherein each of the plurality of conversations comprises a sequence of messages exchanged between a server and a client of the network using the protocol, wherein each message of the sequence of messages comprise one or more fields separated by a field delimiter of the protocol;

extracting, by a computer processor, a plurality of non-alphanumeric tokens from the plurality of conversations, wherein the plurality of non-alphanumeric tokens comprises a non-alphanumeric token associated with a frequency of occurrence in the plurality of conversations;

selecting, based on the frequency of occurrence meeting a pre-determined field delimiter candidate selection criterion, the non-alphanumeric token as a field delimiter candidate;

dividing, by the computer processor and using the field delimiter candidate, each of the plurality of conversations into a plurality of slices;

analyzing, by the computer processor and using a pre-determined field delimiter candidate scoring algorithm, content included in the plurality of slices to:

determine a statistical measure of matched slices for each of the plurality of conversations, wherein the statistical measure of matched slices corresponds to an exact-matched-slices percentage and a prefix-matched-slices percentage that are normalized based on an average number of slices per conversation;

determine a field delimiter candidate score by aggregating the statistical measure of matched slices for all of the plurality of conversations; and

selecting, by the computer processor and based on the field delimiter candidate score associated with the non-alphanumeric token, the non-alphanumeric token as the field delimiter of the protocol.

2. The method of claim 1 , further comprising:

grouping the plurality of slices that are divided from each of the plurality of conversations using the field delimiter of the protocol, into a slice-set for each of the plurality of conversations, wherein the plurality of conversations correspond to a plurality of slice-sets;

extracting, based on a pre-determined key-value delimiter selection criterion, a plurality of longest common prefixes each shared across a portion of the plurality of slice-sets; and

extracting a common trailing token in the plurality of longest common prefixes as the key-value delimiter of the protocol.

3. The method of claim 2 , further comprising:

selecting a first slice from the plurality of sets of slices;

selecting a second slice from each slice-set of a portion of the plurality of slice-sets to generate a group of selected slices, wherein the second slice is selected based on a length of a common prefix shared by the first slice and the second slice; and

analyzing the group of selected slices to identify at least one of the plurality of longest common prefixes.

4. The method of claim 3 ,

wherein the first slice is selected from a first slice-set of the plurality of slice-sets, and

wherein the second slice is selected from each slice-set of the plurality of slice-sets aside from the first slice-set.

5. The method of claim 2 , further comprising:

removing the common trailing token from the plurality of longest common prefixes to identify a keyword of the protocol.

6. The method of claim 2 , further comprising:

identifying a command of the protocol from a slice of the plurality of slices that does not include a key-value delimiter.

7. A system for analyzing a protocol of a network, comprising one or more processors with memory executing:

an acquisition module configured to:

obtain a plurality of conversations from the network, wherein each of the plurality of conversations comprises a sequence of messages exchanged between a server and a client of the network using the protocol, wherein each message of the sequence of messages comprise one or more fields separated by a field delimiter of the protocol;

a field delimiter extractor configured to:

extract a plurality of non-alphanumeric tokens from the plurality of conversations, wherein the plurality of non-alphanumeric tokens comprises a non-alphanumeric token associated with a frequency of occurrence in the plurality of conversations;

select, based on the frequency of occurrence meeting a pre-determined field delimiter candidate selection criterion, the non-alphanumeric token as a field delimiter candidate of a plurality of field delimiter candidates;

divide, using the field delimiter candidate, each of the plurality of conversations into a plurality of slices;

analyze, using a pre-determined field delimiter candidate scoring algorithm, content included in the plurality of slices to:

determine a statistical measure of matched slices for each of the plurality of conversations, wherein the statistical measure of matched slices corresponds to an exact-matched-slices percentage and a prefix-matched-slices percentage that are normalized based on an average number of slices per conversation;

determine a field delimiter candidate score by aggregating the statistical measure of matched slices for all of the plurality of conversations; and

select, based on the field delimiter candidate score associated with the non-alphanumeric token, the non-alphanumeric token from the plurality of field delimiter candidates as the field delimiter of the protocol; and

a repository configured to store the plurality of conversations, the plurality of non-alphanumeric tokens, and the plurality of field delimiter candidates.

8. The system of claim 7 , further comprising the one or more processors executing a key-value delimiter extractor configured to:

group the plurality of slices that are divided from each of the plurality of conversations using the field delimiter of the protocol, into a slice-set for each of the plurality of conversations, wherein the plurality of conversations correspond to a plurality of slice-sets;

extract, based on a pre-determined key-value delimiter selection criterion, a plurality of longest common prefixes each shared across a portion of the plurality of slice-sets; and

extract a common trailing token in the plurality of longest common prefixes as the key-value delimiter of the protocol.

9. The system of claim 8 , wherein the one or more processors executing the key-value delimiter extractor is further configured to:

select a first slice from the plurality of sets of slices;

select a second slice from each slice-set of a portion of the plurality of slice-sets to generate a group of selected slices, wherein the second slice is selected based on a length of a common prefix shared by the first slice and the second slice; and

analyze the group of selected slices to identify at least one of the plurality of longest common prefixes.

10. The system of claim 9 ,

wherein the first slice is selected from a first slice-set of the plurality of slice-sets, and

wherein the second slice is selected from each slice-set of the plurality of slice-sets aside from the first slice-set.

11. The system of claim 8 , further comprising the one or more processors executing a keyword extractor configured to:

remove the common trailing token from the plurality of longest common prefixes to identify a keyword of the protocol.

12. The system of claim 8 , further comprising the one or more processors executing a command extractor configured to:

identify a command of the protocol from a slice of the plurality of slices that does not include a key-value delimiter.

13. A non-transitory computer readable medium embodying instructions for analyzing a protocol of a network, the instructions when executed by a processor comprising functionality for:

obtaining a plurality of conversations from the network, wherein each of the plurality of conversations comprises a sequence of messages exchanged between a server and a client of the network using the protocol, wherein each message of the sequence of messages comprise one or more fields separated by a field delimiter of the protocol;

extracting a plurality of non-alphanumeric tokens from the plurality of conversations, wherein the plurality of non-alphanumeric tokens comprises a non-alphanumeric token associated with a frequency of occurrence in the plurality of conversations;

selecting, based on the frequency of occurrence meeting a pre-determined field delimiter candidate selection criterion, the non-alphanumeric token as a field delimiter candidate;

dividing, using the field delimiter candidate, each of the plurality of conversations into a plurality of slices;

analyzing, using a pre-determined field delimiter candidate scoring algorithm, content included in the plurality of slices to:

determine a statistical measure of matched slices for each of the plurality of conversations, wherein the statistical measure of matched slices corresponds to an exact-matched-slices percentage and a prefix-matched-slices percentage that are normalized based on an average number of slices per conversation;

determine a field delimiter candidate score by aggregating the statistical measure of matched slices for all of the plurality of conversations; and

selecting, based on the field delimiter candidate score associated with the non-alphanumeric token, the non-alphanumeric token as the field delimiter of the protocol.

14. The non-transitory computer readable medium of claim 13 , further comprising:

grouping the plurality of slices that are divided from each of the plurality of conversations using the field delimiter of the protocol, into a slice-set for each of the plurality of conversations, wherein the plurality of conversations correspond to a plurality of slice-sets;

extracting, based on a pre-determined key-value delimiter selection criterion, a plurality of longest common prefixes each shared across a portion of the plurality of slice-sets; and

extracting a common trailing token in the plurality of longest common prefixes as the key-value delimiter of the protocol.

15. The non-transitory computer readable medium of claim 14 , further comprising:

selecting a first slice from the plurality of sets of slices;

selecting a second slice from each slice-set of a portion of the plurality of slice-sets to generate a group of selected slices, wherein the second slice is selected based on a length of a common prefix shared by the first slice and the second slice; and

analyzing the group of selected slices to identify at least one of the plurality of longest common prefixes.

16. The non-transitory computer readable medium of claim 15 ,

wherein the first slice is selected from a first slice-set of the plurality of slice-sets, and

wherein the second slice is selected from each slice-set of the plurality of slice-sets aside from the first slice-set.

17. The non-transitory computer readable medium of claim 14 , further comprising:

removing the common trailing token from the plurality of longest common prefixes to identify a keyword of the protocol.

18. The non-transitory computer readable medium of claim 14 , further comprising:

identifying a command of the protocol from a slice of the plurality of slices that does not include a key-value delimiter.