Automation solutions for event logging and debugging on Kubernetes
Automation solutions for event logging and debugging on KUBERNETES are disclosed. A method for identifying and clustering events on a distributed application cluster includes providing a distributed agent configured to run on the distributed application cluster, the distributed application cluster including a plurality of nodes, wherein at least one distributed application runs on each node of the plurality of nodes, and providing a plurality of edge modules, each edge module being configured to run on a corresponding node of the plurality of nodes. A plurality of data coresets corresponding to each node of the plurality of nodes are tracked via the plurality of edge modules. A unified data coreset is generated by merging the plurality of data coresets and the unified data coreset is transferred to the distributed agent to train an unsupervised model configured to identify and cluster events across the distributed application cluster.
1. A method for identifying and clustering events on a distributed application cluster, the method comprising:
providing a distributed agent configured to run on the distributed application cluster, the distributed application cluster including a plurality of nodes, wherein at least one distributed application runs on each node of the plurality of nodes;
providing a plurality of edge modules, each edge module being configured to run on a corresponding node of the plurality of nodes;
tracking, via the plurality of edge modules, a plurality of data coresets corresponding to each node of the plurality of nodes, wherein each data coreset includes application data captured from the at least one distributed application running on the corresponding node, the captured application data being a representative sample of the node's total application data;
generating a unified data coreset by merging the plurality of data coresets; and
transferring the unified data coreset to the distributed agent to train an unsupervised model configured to identify and cluster events across the distributed application cluster.
2. The method of claim 1 , wherein the unsupervised model is a machine-learning (ML) model.
3. The method of claim 1 , wherein generating the unified data coreset by merging the plurality of data coresets includes transferring the plurality of data coresets to the edge module of a central node of the plurality of nodes.
4. The method of claim 3 , further comprising:
transferring the unified data coreset from the central node to the distributed agent.
5. The method of claim 1 , wherein generating the unified data coreset by merging the plurality of data coresets includes updating the unified data coreset at periodic intervals.
6. The method of claim 1 , further comprising:
receiving a data query at the distributed agent; and
generating the unified data coreset in response to the received data query.
7. The method of claim 1 , further comprising:
deploying, via the plurality of edge modules, at least one Berkeley Packet Filter (BPF) probe and at least one corresponding BPF program on each node of the plurality of nodes.
8. The method of claim 7 , wherein tracking the plurality of data coresets corresponding to each node includes triggering the at least one BPF probe and capturing data via the at least one corresponding BPF program.
9. A system for identifying and clustering events on a distributed application cluster, the system comprising:
at least one memory storing computer-executable instructions; and
at least one processor for executing the computer-executable instructions stored in the memory, wherein the instructions, when executed, instruct the at least one processor to:
provide a distributed agent configured to run on the distributed application cluster, the distributed application cluster including a plurality of nodes, wherein at least one distributed application runs on each node of the plurality of nodes;
provide a plurality of edge modules, each edge module being configured to run on a corresponding node of the plurality of nodes;
track, via the plurality of edge modules, a plurality of data coresets corresponding to each node of the plurality of nodes, wherein each data coreset includes application data captured from the at least one distributed application running on the corresponding node, the captured application data being a representative sample of the node's total application data;
generate a unified data coreset by merging the plurality of data coresets; and
transfer the unified data coreset to the distributed agent to train an unsupervised model configured to identify and cluster events across the distributed application cluster.
10. The system of claim 9 , wherein the unsupervised model is a machine-learning (ML) model.
11. The system of claim 9 , wherein generating the unified data coreset by merging the plurality of data coresets includes transferring the plurality of data coresets to the edge module of a central node of the plurality of nodes.
12. The system of claim 11 , wherein the instructions, when executed, further instruct the at least one processor to:
transfer the unified data coreset from the central node to the distributed agent.
13. The system of claim 9 , wherein generating the unified data coreset by merging the plurality of data coresets includes updating the unified data coreset at periodic intervals.
14. The system of claim 9 , wherein the instructions, when executed, further instruct the at least one processor to:
receive a data query at the distributed agent; and
generate the unified data coreset in response to the received data query.
15. The system of claim 9 , wherein the instructions, when executed, further instruct the at least one processor to:
deploy, via the plurality of edge modules, at least one Berkeley Packet Filter (BPF) probe and at least one corresponding BPF program on each node of the plurality of nodes.
16. The system of claim 15 , wherein tracking the plurality of data coresets corresponding to each node includes triggering the at least one BPF probe and capturing data via the at least one corresponding BPF program.