Method and system for constructing vector databases used for converting free text queries to cyber language queries
A system and method for querying data sources for cybersecurity analysis is presented. The system and method include: receiving security logs from at least one data source, wherein the security logs lack pre-defined schema; generating a schema of the security logs based on at least a type of data of the security logs, wherein the generated schema includes fields of the security logs and values of the fields; embedding field vectors, wherein each field vector is a vector representation of a value of each respective field; embedding value vectors, wherein each value vector is a vector representation of a natural language description of each value in each respective field; and generating a query in a cyber language query, using an AI system, for execution on at least one target data source based, in part, on the generated schema, the embedded field vectors, and the embedded value vectors.
1 . A method for querying data sources for cybersecurity analysis, comprising:
receiving, by a device, security logs from at least one data source, wherein the security logs lack pre-defined schema;
generating, using an artificial intelligence (AI) embedding system, a schema of the security logs based on at least a type of data of the security logs, wherein the generated schema includes fields of the security logs and values of the fields;
embedding, using the AI embedding system, field vectors, wherein each field vector is a vector representation of a value of each respective field;
embedding, using the AI embedding system, value vectors, wherein each value vector is a vector representation of a natural language description of each value in each respective field;
generating a query in a cyber language query, using an AI system; and
executing the generated query on at least one target data source based, in part, on the generated schema, the embedded field vectors, and the embedded value vectors.
2 . The method of claim 1 , further comprising:
generating pairs of NLQs and cyber language queries, wherein the pairs are potentially valid matches between NLQs and the cyber language queries;
validating that the generated cyber language queries are properly matched to the generated NLQs; and
embedding vectors of each of the pairs.
3 . The method of claim 1 , wherein generating the cyber language query, using an AI system further comprises:
identifying fields that are relevant to a natural language query (NLQ) based on a field vector similarity search using the embedded field vectors;
computing values for the identified fields based on actual values that are relevant to the identified fields and the NLQ based on a value vector similarity search using the embedded value vectors;
refining the generated schema based on the identified fields and the computed values;
generating, based on at least the refined schema, a prompt for a Large Language Model (LLM), wherein the LLM is executed by an AI-based query generation system; and
feeding the prompt to the LLM, wherein the LLM executes the prompt using the AI-based query generation system, to convert the NLQ into the cyber language query.
4 . The method of claim 3 , wherein the field vector similarity search further comprises:
comparing, using a semantic distance metric, a feature vector representing the NLQ and each embedded field vector;
identifying at least one field for which the embedded field vector is below a semantic distance threshold from the feature vector representing the NLQ; and
validating the identified at least one field against ground-truth pairs of natural language field descriptions and fields searchable in the cyber language for at least one target data source.
5 . The method of claim 3 , wherein the value vector similarity search further comprises:
comparing, using a semantic distance metric, a feature vector representing the NLQ and each embedded value vector;
identifying at least one value for which the embedded value vector is below a semantic distance threshold from the value vector representing the NLQ; and
validating the at least one value against ground-truth pairs of natural language value descriptions and actual values in the at least one target data source.
6 . The method of claim 3 , wherein refining the schema of the logs based on the identified fields and computed values further comprises:
narrowing the schema to the most pertinent elements of the schema based on the identified fields and computed values.
7 . The method of claim 1 , further comprising:
generating, using the AI embedding system, the natural language descriptions of the received fields, wherein the AI embedding system is configured to gather relevant information on the fields from official documentation and other available data sources to generate the natural language descriptions.
8 . The method of claim 1 , further comprising:
validating syntactical correctness of the generated cyber language query.
9 . The method of claim 1 , wherein the cyber language query is a read-only request to process data stored hierarchically in databases, tables, and columns.
10 . The method of claim 1 , wherein the cyber language is at least Kusto Query Language.
11 . A non-transitory computer-readable medium storing a set of instructions for querying data sources for cybersecurity analysis, the set of instructions comprising:
one or more instructions that, when executed by one or more processors of a device, cause the device to:
receive, by a device, security logs from at least one data source, wherein the security logs lack pre-defined schema;
generate, using an artificial intelligence (AI) embedding system, a schema of the security logs based on at least a type of data of the security logs, wherein the generated schema includes fields of the security logs and values of the fields;
embed, using the AI embedding system, field vectors, wherein each field vector is a vector representation of a value of each respective field;
embed, using the AI embedding system, value vectors, wherein each value vector is a vector representation of a natural language description of each value in each respective field;
generate a query in a cyber language query, using an AI system and
execute the query on at least one target data source based, in part, on the generated schema, the embedded field vectors, and the embedded value vectors.
12 . A system for querying data sources for cybersecurity analysis comprising:
one or more processors; and
a memory including instructions configured to:
receive, by a device, security logs from at least one data source, wherein the security logs lack pre-defined schema;
generate, using an artificial intelligence (AI) embedding system, a schema of the security logs based on at least a type of data of the security logs, wherein the generated schema includes fields of the security logs and values of the fields;
embed, using the AI embedding system, field vectors, wherein each field vector is a vector representation of a value of each respective field;
embed, using the AI embedding system, value vectors, wherein each value vector is a vector representation of a natural language description of each value in each respective field; and
generate a query in a cyber language query, using an AI system; and
execute the query on at least one target data source based, in part, on the generated schema, the embedded field vectors, and the embedded value vectors.
13 . The system of claim 12 , further comprising:
generating pairs of NLQs and cyber language queries, wherein the pairs are potentially valid matches between NLQs and the cyber language queries
validating that the generated cyber language queries are properly matched to the generated NLQs; and
embedding vectors of each of the pairs.
14 . The system of claim 12 , wherein generating the cyber language query, using an AI system further comprises:
identifying fields that are relevant to a natural language query (NLQ) based on a field vector similarity search using the embedded field vectors
computing values for the identified fields based on actual values that are relevant to the identified fields and the NLQ based on a value vector similarity search using the embedded value vectors
refining the generated schema based on the identified fields and the computed values
generating, based on at least the refined schema, a prompt for a Large Language Model (LLM), wherein the LLM is executed by an AI-based query generation system; and
feeding the prompt to the LLM, wherein the LLM executes the prompt using the AI-based query generation system, to convert the NLQ into the cyber language query.
15 . The system of claim 14 , wherein the field vector similarity search further comprises:
comparing, using a semantic distance metric, a feature vector representing the NLQ and each embedded field vector
identifying at least one field for which the embedded field vector is below a semantic distance threshold from the feature vector representing the NLQ; and
validating the identified at least one field against ground-truth pairs of natural language field descriptions and fields searchable in the cyber language for at least one target data source.
16 . The system of claim 14 , wherein the value vector similarity search further comprises:
comparing, using a semantic distance metric, a feature vector representing the NLQ and each embedded value vector
identifying at least one value for which the embedded value vector is below a semantic distance threshold from the value vector representing the NLQ; and
validating the at least one value against ground-truth pairs of natural language value descriptions and actual values in the at least one target data source.
17 . The system of claim 14 , wherein refining the schema of the logs based on the identified fields and computed values further comprises:
narrowing the schema to the most pertinent elements of the schema based on the identified fields and computed values.
18 . The system of claim 12 , further comprising:
generating, using the AI embedding system, the natural language descriptions of the received fields, wherein the AI embedding system is configured to gather relevant information on the fields from official documentation and other available data sources to generate the natural language descriptions.
19 . The system of claim 12 , further comprising:
validating syntactical correctness of the generated cyber language query.
20 . The system of claim 12 , wherein the cyber language query is a read-only request to process data stored hierarchically in databases, tables, and columns.
21 . The system of claim 12 , wherein the cyber language is at least Kusto Query Language.