System and method for network data compression

1. A method of compressing data in a network, the data comprising a plurality of packets corresponding to a plurality of headers and a plurality of payloads, a packet of the plurality of packets having a header of the plurality of headers and a payload of the plurality of payloads, each of the plurality of headers comprising a plurality of header fields, the method comprising:

generating, using a computer, a matrix based on the plurality of packets, the matrix comprising a plurality of rows each corresponding to one of the plurality of packets and a plurality of structured columns each corresponding to one of the plurality of header fields, the matrix further comprising an unstructured column having a plurality of row elements each comprising one of the plurality of payloads;

determining, using the computer, an intra-packet compression plan based on historical network data, the intra-packet compression plan configured to reorder and divide the plurality of structured columns in the matrix into a plurality of groups;

determining, using the computer, an inter-packet compression plan based on historical network data, the inter-packet compression plan configured to reorder and classify the plurality of rows in the matrix based on a plurality of hierarchical categories in a classification tree determined using the historical network data, wherein the classification tree is determined by:

composing a full classification tree having a plurality of nodes based on the plurality of header fields, each of the plurality of nodes representing payload classification based on a specific value combination of the plurality of header fields;

identifying a hierarchical category of the plurality of hierarchical categories from the full classification tree, the hierarchical category comprising one or more nodes in the plurality of nodes identified based on minimizing a first cost function while applying the inter-packet compression plan to the historical network data; and

including the hierarchical category in the classification tree; and

performing, using the computer, inter-packet compression of the plurality of payloads based on the inter-packet compression plan; and

performing, using the computer, intra-packet compression of the plurality of headers based on the intra-packet compression plan.

2. The method of claim 1 , further comprising:

obtaining the plurality of packets in real time in a compression phase subsequent to determining the intra-packet compression plan and the inter-packet compression plan based on the historical network data during a training phase,

wherein the intra-packet compression plan is determined during the training phase based on minimizing a second cost function while applying the intra-packet compression plan to the historical network data,

wherein performing the intra-packet compression comprises performing first online compression in real time for the plurality of headers using a plurality of header compressors corresponding to the plurality of groups, wherein the first online compression is performed concurrently for each of the plurality of groups using a corresponding one of the plurality of header compressors,

wherein performing the inter-packet compression comprises performing second online compression in real time for the plurality of payloads using a plurality of payload compressors corresponding to a portion of the plurality of hierarchical categories based on a top portion of the classification tree, wherein the second online compression is performed concurrently for each hierarchical category in the portion of the plurality of hierarchical categories using a corresponding one of the plurality of payload compressors.

3. A non-transitory computer readable medium, embodying instructions executable by the computer to perform method steps for compressing data in a network, the data comprising a plurality of packets corresponding to a plurality of headers and a plurality of payloads, a packet of the plurality of packets having a header of the plurality of headers and a payload of the plurality of payloads, each of the plurality of headers comprising a plurality of header fields, the instructions comprising functionality for:

generating, a matrix based on the plurality of packets, the matrix comprising a plurality of rows each corresponding to one of the plurality of packets and a plurality of structured columns each corresponding to one of the plurality of header fields;

determining an intra-packet compression plan based on historical network data, the intra-packet compression plan configured to reorder and divide the plurality of structured columns in the matrix into a plurality of groups,

determining an inter-packet compression plan based on historical network data, the inter-packet compression plan configured to reorder and classify the plurality of rows in the matrix based on a plurality of hierarchical categories in a classification tree determined using the historical network data, wherein the classification tree is determined by:

composing a full classification tree having a plurality of nodes based on the plurality of header fields, each of the plurality of nodes representing payload classification based on a specific value combination of the plurality of header fields;

identifying a hierarchical category of the plurality of hierarchical categories from the full classification tree, the hierarchical category comprising one or more nodes in the plurality of nodes identified based on minimizing a first cost function while applying the inter-packet compression plan to the historical network data; and

including the hierarchical category in the classification tree; and

performing inter-packet compression of the plurality of payloads based on the inter-packet compression plan; and

performing, intra-packet compression of the plurality of headers based on the intra-packet compression plan.

4. The method of claim 2 , further comprising:

performing at least one selected from a group consisting of the intra-packet compression and the inter-packet compression of the plurality of packets while the plurality of packets are being transferred by a network node of the network.

5. The method of claim 2 , further comprising:

performing at least one selected from a group consisting of the intra-packet compression and the inter-packet compression of the plurality of packets subsequent to the plurality of packets being collected by a data repository of the network.

6. The method of claim 2 , further comprising:

comparing a cumulative compression ratio and a pre-determined threshold to generate a result, wherein the cumulative compression ratio is based on cumulatively performing at least one selected from a group consisting of the intra-packet compression and the inter-packet compression subsequent to the training phase; and

adjusting at least one selected from a group consisting of the intra-packet compression plan and the inter-packet compression plan to generate an adjusted compression plan according to the result.

7. The method of claim 6 , wherein the pre-determined threshold is a function of time period subsequent to the training phase.

8. The method of claim 6 ,

wherein adjusting at least one selected from a group consisting of the intra-packet compression plan and the inter-packet compression plan comprises adjusting the classification tree.

9. The method of claim 6 , further comprising:

determining an optimal time to replace at least one selected from a group consisting of the intra-packet compression plan and the inter-packet compression plan by the adjusted compression plan.

10. The non-transitory computer readable medium of claim 3 , the instructions further comprising functionality for:

obtaining the plurality of packets in real time in a compression phase subsequent to determining the intra-packet compression plan and the inter-packet compression plan based on the historical network data during a training phase,

wherein the intra-packet compression plan is determined during the training phase based on minimizing a second cost function while applying the intra-packet compression plan to the historical network data,

wherein performing the intra-packet compression comprises performing first online compression in real time for the plurality of headers using a plurality of header compressors corresponding to the plurality of groups, wherein the first online compression is performed concurrently for each of the plurality of groups using a corresponding one of the plurality of header compressors,

wherein performing the inter-packet compression comprises performing second online compression in real time for the plurality of payloads using a plurality of payload compressors corresponding to a portion of the plurality of hierarchical categories based on a top portion of the classification tree, wherein the second online compression is performed concurrently for each hierarchical category in the portion of the plurality of hierarchical categories using a corresponding one of the plurality of payload compressors.

11. The non-transitory computer readable medium of claim 10 , the instructions further comprising functionality for:

performing at least one selected from a group consisting of the intra-packet compression and the inter-packet compression of the plurality of packets while the plurality of packets are being transferred by a network node of the network.

12. The non-transitory computer readable medium of claim 10 , the instructions further comprising functionality for:

performing at least one selected from a group consisting of the intra-packet compression and the inter-packet compression of the plurality of packets subsequent to the plurality of packets being collected by a data repository of the network.

13. The non-transitory computer readable medium of claim 10 , the instructions further comprising functionality for:

comparing a cumulative compression ratio and a pre-determined threshold to generate a result, wherein the cumulative compression ratio is based on cumulatively performing at least one selected from a group consisting of the intra-packet compression and the inter-packet compression subsequent to the training phase; and

adjusting at least one selected from a group consisting of the intra-packet compression plan and the inter-packet compression plan to generate an adjusted compression plan according to the result.

14. The non-transitory computer readable medium of claim 13 , wherein the predetermined threshold is a function of time period subsequent to the training phase.

15. The non-transitory computer readable medium of claim 13 , wherein adjusting at least one selected from a group consisting of the intra-packet compression plan and the inter-packet compression plan comprises adjusting the classification tree.

16. The non-transitory computer readable medium of claim 13 , the instructions further comprising functionality for:

determining an optimal time to replace at least one selected from a group consisting of the intra-packet compression plan and the inter-packet compression plan by the adjusted compression plan.