Methods for optimizing transport of unknown and changing unstructured log data and devices thereof

1 . A method, implemented by a network traffic management system comprising one or more network traffic management apparatuses, edge devices, client devices, or server devices, comprising:

receiving a query from a log processing device configured to process target unstructured log data, the target unstructured log data comprising information describing events that have occurred in a network, a network application, or a network device;

retrieving, responsive to the received query, one or more entries associated with the target unstructured log data and stored in a log catalog storage;

structuring, via a log stream batcher and structurer, the retrieved one or more entries into sorted structured batches comprising structured data and sorted unstructured batches without structured data, based on a current configuration and producing corresponding current batch metadata, wherein the current configuration comprises a set of patterns and a set of instructions for the structuring for sorting the structured data in the sorted structured batches across one or more columns, and wherein the current batch metadata comprises:

identification of elements of the current configuration used to produce the sorted structured batches; and

compressed sizes of the sorted structured batches and the sorted unstructured batches;

converting the sorted structured batches into columnar-oriented batches;

compressing the columnar-oriented batches;

evaluating, via a compression rate evaluator, the compressed columnar-oriented batches by comparing a compression ratio of the compressed sizes of the sorted structured batches and the sorted unstructured batches using the current batch metadata thereby producing a compression rate;

receiving, via an online optimizer from the compression rate evaluator, the compression rate, and the current batch metadata;

determining, via the online optimizer, an optimized configuration for the retrieved one or more entries based on the compression rate and the current batch metadata;

sending, via the online optimizer to the log stream batcher and structurer, the optimized configuration; and

repeating the structuring and subsequent steps using the optimized configuration.

2 . The method of claim 1 , further comprising:

decompressing the compressed columnar-oriented batches into columnar-oriented batches containing the log data; and

reconstructing the target unstructured log data using the columnar-oriented batches containing the log data and the set of patterns.

3 . The method of claim 1 , further comprising:

receiving, via a log analyzer and catalog from the compression rate evaluator, a sample of the sorted structured batches along with the current batch metadata and compression rate for further analysis and cataloging;

receiving, via the online optimizer from the log analyzer and catalog, data corresponding to the further analysis and cataloging;

determining, via the online optimizer, a further optimized configuration for the sample based on the further analysis and cataloging;

sending, via the online optimizer to the log stream batcher and structurer, the further optimized configuration; and

repeating the structuring and subsequent steps using the further optimized configuration.

4 . The method of claim 3 , wherein the further analysis is guided by a user-defined objection function (UDOF).

5 . The method of claim 4 , wherein the UDOF enables a user to define a tradeoff between optimizing transport and optimizing a stability of schemas inferred from the further analysis performed by the log analyzer and catalog.

6 . An apparatus, comprising memory comprising programmed instructions stored in the memory and one or more processors configured to be capable of executing the programmed instructions stored in the memory to:

receive a query from a log processing device configured to process target unstructured log data, the target unstructured log data comprising information describing events that have occurred in a network, a network application, or a network device;

retrieve, responsive to the received query, one or more entries associated with the target unstructured log data and stored in a log catalog storage;

structure, via a log stream batcher and structurer, the retrieved one or more entries into sorted structured batches comprising structured data and sorted unstructured batches without structured data, based on a current configuration and producing corresponding current batch metadata, wherein the current configuration comprises a set of patterns and a set of instructions for the structuring for sorting the structured data in the sorted structured batches across one or more columns, and wherein the current batch metadata comprises:

identification of elements of the current configuration used to produce the sorted structured batches; and

compressed sizes of the sorted structured batches and the sorted unstructured batches;

convert the sorted structured batches into columnar-oriented batches;

compress the columnar-oriented batches;

evaluate, via a compression rate evaluator, the compressed columnar-oriented batches by comparing a compression ratio of the compressed sizes of the sorted structured batches and the sorted unstructured batches using the current batch metadata thereby producing a compression rate;

receive, via an online optimizer from the compression rate evaluator, the compression rate, and the current batch metadata;

determine, via the online optimizer, an optimized configuration for the retrieved one or more entries based on the compression rate and the current batch metadata;

send, via the online optimizer to the log stream batcher and structurer, the optimized configuration; and

repeat the structuring and subsequent steps using the optimized configuration.

7 . The apparatus of claim 6 , wherein the one or more processors are further configured to be capable of executing the programmed instructions stored in the memory to:

decompressing the compressed columnar-oriented batches into columnar-oriented batches containing the log data; and

reconstructing the target unstructured log data using the columnar-oriented batches containing the log data and the set of patterns.

8 . The apparatus of claim 6 , wherein the one or more processors are further configured to be capable of executing the programmed instructions stored in the memory to:

receiving, via a log analyzer and catalog from the compression rate evaluator, a sample of the sorted structured batches along with the current batch metadata and compression rate for further analysis and cataloging;

receiving, via the online optimizer from the log analyzer and catalog, data corresponding to the further analysis and cataloging;

determining, via the online optimizer, a further optimized configuration for the sample based on the further analysis and cataloging;

sending, via the online optimizer to the log stream batcher and structurer, the further optimized configuration; and

repeating the structuring and subsequent steps using the further optimized configuration.

9 . The apparatus of claim 8 , wherein the further analysis is guided by a user-defined objection function (UDOF).

10 . The apparatus of claim 9 , wherein the UDOF enables a user to define a tradeoff between optimizing transport and optimizing a stability of schemas inferred from the further analysis performed by the log analyzer and catalog.

11 . A non-transitory computer readable medium having stored thereon instructions, comprising executable code which when executed by one or more processors, causes the one or more processors to:

receive a query from a log processing device configured to process target unstructured log data, the target unstructured log data comprising information describing events that have occurred in a network, a network application, or a network device;

retrieve, responsive to the received query, one or more entries associated with the target unstructured log data and stored in a log catalog storage;

structure, via a log stream batcher and structurer, the retrieved one or more entries into sorted structured batches comprising structured data and sorted unstructured batches without structured data, based on a current configuration and producing corresponding current batch metadata, wherein the current configuration comprises a set of patterns and a set of instructions for the structuring for sorting the structured data in the sorted structured batches across one or more columns, and wherein the current batch metadata comprises:

identification of elements of the current configuration used to produce the sorted structured batches; and

compressed sizes of the sorted structured batches and the sorted unstructured batches;

convert the sorted structured batches into columnar-oriented batches;

compress the columnar-oriented batches;

evaluate, via a compression rate evaluator, the compressed columnar-oriented batches by comparing a compression ratio of the compressed sizes of the sorted structured batches and the sorted unstructured batches using the current batch metadata thereby producing a compression rate;

receive, via an online optimizer from the compression rate evaluator, the compression rate, and the current batch metadata;

determine, via the online optimizer, an optimized configuration for the retrieved one or more entries based on the compression rate and the current batch metadata;

send, via the online optimizer to the log stream batcher and structurer, the optimized configuration; and

repeat the structuring and subsequent steps using the optimized configuration.

12 . The non-transitory computer readable medium of claim 11 , wherein the one or more processors are further configured to be capable of executing the programmed instructions stored in the memory to:

decompressing the compressed columnar-oriented batches into columnar-oriented batches containing the log data; and

reconstructing the target unstructured log data using the columnar-oriented batches containing the log data and the set of patterns.

13 . The non-transitory computer readable medium of claim 11 , wherein the one or more processors are further configured to be capable of executing the programmed instructions stored in the memory to:

receiving, via a log analyzer and catalog from the compression rate evaluator, a sample of the sorted structured batches along with the current batch metadata and compression rate for further analysis and cataloging;

receiving, via the online optimizer from the log analyzer and catalog, data corresponding to the further analysis and cataloging;

determining, via the online optimizer, a further optimized configuration for the sample based on the further analysis and cataloging;

sending, via the online optimizer to the log stream batcher and structurer, the further optimized configuration; and

repeating the structuring and subsequent steps using the further optimized configuration.

14 . The non-transitory computer readable medium of claim 13 , wherein the further analysis is guided by a user-defined objection function (UDOF).

15 . The non-transitory computer readable medium of claim 14 , wherein the UDOF enables a user to define a tradeoff between optimizing transport and optimizing a stability of schemas inferred from the further analysis performed by the log analyzer and catalog.

16 . A network traffic management system, comprising one or more traffic management apparatuses, server devices, or client devices, the network traffic management system comprising memory comprising programmed instructions stored thereon and one or more processors configured to be capable of executing the stored programmed instructions to:

receive a query from a log processing device configured to process target unstructured log data, the target unstructured log data comprising information describing events that have occurred in a network, a network application, or a network device;

retrieve, responsive to the received query, one or more entries associated with the target unstructured log data and stored in a log catalog storage;

structure, via a log stream batcher and structurer, the retrieved one or more entries into sorted structured batches comprising structured data and sorted unstructured batches without structured data, based on a current configuration and producing corresponding current batch metadata, wherein the current configuration comprises a set of patterns and a set of instructions for the structuring for sorting the structured data in the sorted structured batches across one or more columns, and wherein the current batch metadata comprises:

identification of elements of the current configuration used to produce the sorted structured batches; and

compressed sizes of the sorted structured batches and the sorted unstructured batches;

convert the sorted structured batches into columnar-oriented batches;

compress the columnar-oriented batches;

evaluate, via a compression rate evaluator, the compressed columnar-oriented batches by comparing a compression ratio of the compressed sizes of the sorted structured batches and the sorted unstructured batches using the current batch metadata thereby producing a compression rate;

receive, via an online optimizer from the compression rate evaluator, the compression rate, and the current batch metadata;

determine, via the online optimizer, an optimized configuration for the retrieved one or more entries based on the compression rate and the current batch metadata;

send, via the online optimizer to the log stream batcher and structurer, the optimized configuration; and

repeat the structuring and subsequent steps using the optimized configuration.

17 . The network traffic management system of claim 16 , wherein the one or more processors are further configured to be capable of executing the programmed instructions stored in the memory to:

decompressing the compressed columnar-oriented batches into columnar-oriented batches containing the log data; and

reconstructing the target unstructured log data using the columnar-oriented batches containing the log data and the set of patterns.

18 . The network traffic management system of claim 16 , wherein the one or more processors are further configured to be capable of executing the programmed instructions stored in the memory to:

receiving, via a log analyzer and catalog from the compression rate evaluator, a sample of the sorted structured batches along with the current batch metadata and compression rate for further analysis and cataloging;

receiving, via the online optimizer from the log analyzer and catalog, data corresponding to the further analysis and cataloging;

determining, via the online optimizer, a further optimized configuration for the sample based on the further analysis and cataloging;

sending, via the online optimizer to the log stream batcher and structurer, the further optimized configuration; and

repeating the structuring and subsequent steps using the further optimized configuration.

19 . The network traffic management system of claim 18 , wherein the further analysis is guided by a user-defined objection function (UDOF).

20 . The network traffic management system of claim 19 , wherein the UDOF enables a user to define a tradeoff between optimizing transport and optimizing a stability of schemas inferred from the further analysis performed by the log analyzer and catalog.