Hybrid log viewer with thin memory usage
A method is disclosed, comprising: storing, in a memory of a first computing device, a log filter list that identifiers a plurality of log filters, each of the plurality of log filters being executed on a respective one of a plurality of nodes in a distributed computing system; removing a first log message from a data structure that is stored in the memory of the first computing device, the first log message being obtained by the first computing device from a first one of the plurality of log filters that is associated with a first log source; detecting whether the first log message includes a placeholder message that is generated by the first log filter while the first log source is in a paused state; and selecting the first log message for further processing, when the first log message does not include a placeholder message.
1. A method for collecting log data in a distributed computing system, comprising:
storing, in a memory of a first computing device, a log filter list that identifiers a plurality of log filters, each of the plurality of log filters being executed on a respective one of a plurality of nodes in the distributed computing system;
removing a first log message from a data structure that is stored in the memory of the first computing device, the first log message being obtained by the first computing device from a first one of the plurality of log filters that is associated with a first log source, the first log message being removed from the data structure based on having a least recent timestamp among a plurality of log messages that are stored in the data structure;
detecting whether the first log message includes a placeholder message that is generated by the first log filter while the first log source is in a paused state;
discarding the first log message, when the first log message includes a placeholder message; and
selecting the first log message for further processing, when the first log message does not include a placeholder message.
2. The method of claim 1 , further comprising obtaining a second log message from the first log filter and including the second log message into the data structure to replace the first log message.
3. The method of claim 1 , wherein detecting whether the first log message is a placeholder message includes detecting whether a first timestamp of the first log message is less than a second timestamp of a second log message that is removed from the data structure before the first log message.
4. The method of claim 1 , wherein the data structure is configured to store no more than a predetermined number of log messages that are generated by any given one of the plurality of log filters.
5. The method of claim 1 , further comprising:
retrieving, by the first computing device, a second log message from the data structure, the second log message being obtained by the first computing device from a second one of the plurality of log filters, the second log filter being associated with a second log source;
detecting, by the first computing device, whether the second log message includes an end-of-stream (EOS) message indicating that an execution of the second log source has been completed; and
removing the second log filter from the log filter list when the second log message includes an EOS message.
6. The method of claim 1 , wherein selecting the first log message for further processing includes at least one of writing the first log message to an output stream or storing the first log message in a predetermined location in the memory of the first computing device.
7. The method of claim 1 , wherein selecting the first log message for further processing includes transmitting the first log message from the first computing device to a second computing device.
8. A system comprising:
a memory; and
at least one processor operatively coupled to the memory, the at least one processor being configured to:
store, in the memory, a log filter list that identifiers a plurality of log filters, each of the plurality of log filters being executed on a respective one of a plurality of nodes in a distributed computing system;
remove a first log message from a data structure that is stored in the memory, the first log message being obtained from a first one of the plurality of log filters that is associated with a first log source, the first log message being removed from the data structure based on having a least recent timestamp among a plurality of log messages that are stored in the data structure;
detect whether the first log message includes a placeholder message that is generated by the first log filter while the first log source is in a paused state;
discard the first log message, when the first log message includes a placeholder message; and
select the first log message for further processing, when the first log message does not include a placeholder message.
9. The system of claim 8 , wherein the at least one processor is further configured to obtain a second log message from the first log filter and including the second log message into the data structure to replace the first log message.
10. The system of claim 8 , wherein detecting whether the first log message is a placeholder message includes detecting whether a first timestamp of the first log message is less than a second timestamp of a second log message that is removed from the data structure before the first log message.
11. The system of claim 8 , wherein the data structure is configured to store no more than a predetermined number of log messages that are generated by any given one of the plurality of log filters.
12. The system of claim 8 , wherein the at least one processor is further configured to:
retrieve a second log message from the data structure, the second log message being obtained from a second one of the plurality of log filters, the second log filter being associated with a second log source;
detect whether the second log message includes an end-of-stream (EOS) message indicating that an execution of the second log source has been completed; and
remove the second log filter from the log filter list when the second log message includes an EOS message.
13. The system of claim 8 , wherein selecting the first log message for further processing includes at least one of writing the first log message to an output stream or storing the first log message in a predetermined location in the memory.
14. The system of claim 8 , wherein selecting the first log message for further processing includes transmitting the first log message a remote computing device.
15. A non-transitory computer-readable medium configured to store one or more processor-executable instructions, which when executed by at least one processor cause the at least one processor to perform the operations of:
storing, in a memory that is operatively coupled to the at least one processor, a log filter list that identifiers a plurality of log filters, each of the plurality of log filters being executed on a respective one of a plurality of nodes in a distributed computing system;
removing a first log message from a data structure that is stored in the memory, the first log message being obtained from a first one of the plurality of log filters that is associated with a first log source, the first log message being removed from the data structure based on having a least recent timestamp among a plurality of log messages that are stored in the data structure;
detecting whether the first log message includes a placeholder message that is generated by the first log filter while the first log source is in a paused state;
discarding the first log message, when the first log message includes a placeholder message; and
selecting the first log message for further processing, when the first log message does not include a placeholder message.
16. The non-transitory computer-readable medium of claim 15 , wherein the one or more processor-executable instructions, when executed by the at least one processor, further cause the at least one processor to obtain a second log message from the first log filter and including the second log message into the data structure to replace the first log message.
17. The non-transitory computer-readable medium of claim 15 , wherein detecting whether the first log message is a placeholder message includes detecting whether a first timestamp of the first log message is less than a second timestamp of a second log message that is removed from the data structure before the first log message.
18. The non-transitory computer-readable medium of claim 15 , wherein the data structure is configured to store no more than a predetermined number of log messages that are generated by any given one of the plurality of log filters.
19. The non-transitory computer-readable medium of claim 15 , wherein the one or more processor-executable instructions, when executed by the at least one processor, further cause the at least one processor to perform the operations of:
retrieving a second log message from the data structure, the second log message being obtained from a second one of the plurality of log filters, the second log filter being associated with a second log source;
detecting whether the second log message includes an end-of-stream (EOS) message indicating that an execution of the second log source has been completed; and
removing the second log filter from the log filter list when the second log message includes an EOS message.
20. The non-transitory computer-readable medium of claim 15 , wherein selecting the first log message for further processing includes at least one of writing the first log message to an output stream or storing the first log message in a predetermined location in the memory.