Dynamic data log mining for issue resolution
A computer-implemented system and method for processing log data in a multi-tenant software environment processes logs from distributed software modules through an automated pipeline. The system implements developer-configured event definitions to identify discrete events using start and end message pairs, while extracting reference text patterns to correlate logs across different modules. A hybrid database structure combines database and vector database capabilities to maintain both filtered data and complete textual context, enabling semantic processing of natural language queries. The system performs automated cross-module correlation through language model processing that extracts structured information based on developer-specified parameters. Two-stage query processing uses language embeddings for semantic matching and automated database query generation to provide contextualized responses. The system maintains tenant isolation through boundary validation while enabling resource optimization across distributed components, improving log processing efficiency and reducing issue resolution time in complex multi-tenant environments.
1 . A system for processing log data in a multi-tenant environment, comprising:
at least one processor;
at least one graphics processing unit (GPU); and
memory comprising instructions that, when executed by the at least one processor and the GPU, causes the at least one processor and the GPU to perform operations to:
receive log data from a plurality of distributed software components, wherein each software component generates logs for different types of operations;
process the log data using developer-configured event definitions that specify start messages and end messages for identifying discrete events;
correlate the identified discrete events across the distributed software components by:
extracting reference text patterns specified in the developer-configured event definitions, identifying relationships between logs from different components based on the extracted reference text patterns, and combining related logs into unified event records;
storing the unified event records in a hybrid database structure that maintains both filtered data and complete textual context, wherein the hybrid database structure comprises a vector database configured for fast semantic comparison of log-specific textual information and to handle billions of records while maintaining quick query processing, and is integrated with SQL tables for hybrid structured/semantic querying, and wherein each unified event record is stored together with an embedding for semantic comparison; and
process natural language queries by:
generating a query embedding for each natural language query using a GPU-accelerated neural network;
performing semantic matching by executing an approximate-nearest-neighbor search over a graph-based vector index using the query embedding;
automatically generating SQL queries based on the semantic matching; and
providing responses based on the unified event records and contextual textual data retrieved from the hybrid database structure.
2 . The system of claim 1 , the instructions to process the log data further comprising instructions that, when executed by the at least one processor, causes the at least one processor to perform operations to:
validate tenant boundaries for each log entry; and
maintain tenant-specific context across the distributed software components.
3 . The system of claim 1 , the instructions to correlate the identified discrete events further comprising instructions that, when executed by the at least one processor, causes the at least one processor to perform operations to:
use large language models to extract structured information based on developer-specified function parameters; and
maintain chronological relationships between events across the distributed software components.
4 . The system of claim 1 , wherein the hybrid database structure comprises:
a SQL database for storing structured event data; and
a vector database for enabling semantic comparison of textual information.
5 . The system of claim 1 , the memory further comprising instructions that, when executed by the at least one processor, causes the at least one processor to perform operations to:
automatically generate event configurations from developer-provided declarative configurations; and
dynamically update processing rules based on the generated event configurations.
6 . The system of claim 1 , the instructions to process natural language queries further comprising instructions that, when executed by the at least one processor, causes the at least one processor to perform operations to:
perform two-stage language model processing comprising:
first stage processing to identify relevant event types, and
second stage processing to extract specific filters and generate detailed responses.
7 . The system of claim 1 , the memory further comprising instructions that, when executed by the at least one processor, causes the at least one processor to perform operations to:
maintain plain text versions of logs for historical analysis; and
track relationships between independently stored logs across different system components.
8 . At least one non-transitory machine-readable medium comprising instructions for processing log data in a multi-tenant environment that, when executed by at least one processor, causes the at least one processor to perform operations to:
receive log data from a plurality of distributed software components, wherein each software component generates logs for different types of operations;
process the log data using developer-configured event definitions that specify start messages and end messages for identifying discrete events;
correlate the identified discrete events across the distributed software components by:
extracting reference text patterns specified in the developer-configured event definitions, identifying relationships between logs from different components based on the extracted reference text patterns, and combining related logs into unified event records;
storing the unified event records in a hybrid database structure that maintains both filtered data and complete textual context, wherein the hybrid database structure comprises a vector database configured for fast semantic comparison of log-specific textual information and to handle billions of records while maintaining quick query processing, and is integrated with SQL tables for hybrid structured/semantic querying, and wherein each unified event record is stored together with an embedding for semantic comparison; and
process natural language queries by:
generating a query embedding for each natural language query using a graphics processing unit (GPU)-accelerated neural network;
performing semantic matching by executing an approximate-nearest-neighbor search over a graph-based vector index using the query embedding;
automatically generating SQL queries based on the semantic matching; and
providing responses based on the unified event records and contextual textual data retrieved from the hybrid database structure.
9 . The at least one non-transitory machine-readable medium of claim 8 , the instructions to process the log data further comprising instructions that, when executed by the at least one processor, causes the at least one processor to perform operations to:
validate tenant boundaries for each log entry; and
maintain tenant-specific context across the distributed software components.
10 . The at least one non-transitory machine-readable medium of claim 8 , the instructions to correlate the identified discrete events further comprising instructions that, when executed by the at least one processor, causes the at least one processor to perform operations to:
use large language models to extract structured information based on developer-specified function parameters; and
maintain chronological relationships between events across the distributed software components.
11 . The at least one non-transitory machine-readable medium of claim 8 , wherein the hybrid database structure comprises:
a SQL database for storing structured event data; and
a vector database for enabling semantic comparison of textual information.
12 . The at least one non-transitory machine-readable medium of claim 8 , further comprising instructions that, when executed by the at least one processor, causes the at least one processor to perform operations to:
automatically generate event configurations from developer-provided declarative configurations; and
dynamically update processing rules based on the generated event configurations.
13 . The at least one non-transitory machine-readable medium of claim 8 , the instructions to process natural language queries further comprising instructions that, when executed by the at least one processor, causes the at least one processor to perform operations to:
perform two-stage language model processing comprising:
first stage processing to identify relevant event types, and
second stage processing to extract specific filters and generate detailed responses.
14 . The at least one non-transitory machine-readable medium of claim 8 , further comprising instructions that, when executed by the at least one processor, causes the at least one processor to perform operations to:
maintain plain text versions of logs for historical analysis; and
track relationships between independently stored logs across different system components.
15 . A computer-implemented method for processing log data in a multi-tenant environment, comprising:
receiving log data from a plurality of distributed software components, wherein each software component generates logs for different types of operations;
processing the log data using developer-configured event definitions that specify start messages and end messages for identifying discrete events;
correlating the identified discrete events across the distributed software components by:
extracting reference text patterns specified in the developer-configured event definitions, identifying relationships between logs from different components based on the extracted reference text patterns, and combining related logs into unified event records;
storing the unified event records in a hybrid database structure that maintains both filtered data and complete textual context, wherein the hybrid database structure comprises a vector database configured for fast semantic comparison of log-specific textual information and to handle billions of records while maintaining quick query processing, and is integrated with SQL tables for hybrid structured/semantic querying, and wherein each unified event record is stored together with an embedding for semantic comparison; and
processing natural language queries by:
generating a query embedding for each natural language query using a graphics processing unit (GPU)-accelerated neural network;
performing semantic matching by executing an approximate-nearest-neighbor search over a graph-based vector index using the query embedding;
automatically generating SQL queries based on the semantic matching; and
providing responses based on the unified event records and contextual textual data retrieved from the hybrid database structure.
16 . The method of claim 15 , wherein processing the log data further comprises:
validating tenant boundaries for each log entry; and
maintaining tenant-specific context across the distributed software components.
17 . The method of claim 15 , wherein correlating the identified discrete events comprises:
using large language models to extract structured information based on developer-specified function parameters; and
maintaining chronological relationships between events across the distributed software components.
18 . The method of claim 15 , wherein the hybrid database structure comprises:
a SQL database for storing structured event data; and
a vector database for enabling semantic comparison of textual information.
19 . The method of claim 15 , further comprising:
automatically generating event configurations from developer-provided declarative configurations; and
dynamically updating processing rules based on the generated event configurations.
20 . The method of claim 15 , wherein processing natural language queries comprises:
performing two-stage language model processing comprising:
first stage processing to identify relevant event types, and
second stage processing to extract specific filters and generate detailed responses.