Connecting natural and security language in the embedding space for better threat hunting and incident response
Methods and apparatuses for improving the speed, quality, and relevance of automated responses provided by a question answering system for security data are described. The question answering system may generate and utilize a large language model that is trained to combine the language of security data, such as the language found in security logs and alerts, with natural language text. Given an input prompt (or a search query) from an end user of the question answering system, the question answering system may identify relevant content from the security data and display a response based on the relevant content. The question answering system may allow the end user of the question answering system to query security logs using natural language text without requiring the end user to provide a structured query and without requiring the security data be parsed and ingested into a database system.
1 . A system, comprising:
a storage device configured to store a large language model; and
at least one processor in communication with the storage device that is configured to:
receive security data, the security data includes a plurality of log lines;
map a first log line of the plurality of log lines to an event identifier associated with a type of security event;
set a window size for the first log line of the plurality of log lines based on the event identifier for the first log line;
partition the first log line based on the window size;
determine a prompt for generating a plurality of natural language descriptions corresponding with the plurality of log lines;
generate, using the prompt, the plurality of natural language descriptions corresponding with the plurality of log lines;
determine a plurality of template identifiers based on similarity between natural language descriptions of the plurality of natural language descriptions;
generate, using the plurality of template identifiers, positive pairings and negative pairings of the plurality of natural language descriptions corresponding with the plurality of log lines;
train the large language model using the positive pairings and the negative pairings;
generate, using the large language model, a response that identifies a cyber attack; and
perform a security risk mitigation action based on the response.
2 . The system of claim 1 , wherein:
the positive pairings include a first pairing of the plurality of natural language descriptions corresponding with a first log line and a second log line of the plurality of log lines;
the negative pairings include a second pairing of the plurality of natural language descriptions corresponding with a third log line and a fourth log line of the plurality of log lines; and
the at least one processor is configured to fine-tune the large language model such that the large language model generates similar embeddings with at most a first embedding distance given the first pairing and generates different embeddings with at least a second embedding distance greater than the first embedding distance given the second pairing.
3 . The system of claim 2 , wherein:
the first embedding distance corresponds with a cosine similarity distance; and
the prompt comprises natural language text that is determined based on a type of security document storing the security data.
4 . The system of claim 1 , wherein:
the positive pairings include a first pairing corresponding with a first pair of log lines of the plurality of log lines;
the negative pairings include a second pairing corresponding with a second pair of log lines of the plurality of log lines; and
the at least one processor is configured to fine-tune the large language model such that the first pairing are given similar embeddings within a first embedding distance while the second pair are given different embeddings with a second embedding distance greater than the first embedding distance.
5 . The system of claim 1 , wherein:
the at least one processor is configured to fine-tune the large language model to generate embeddings given the positive pairings and the negative pairings as inputs to the large language model.
6 . The system of claim 1 , wherein:
the at least one processor is configured to generate, using the plurality of template identifiers, a first grouping of log lines out of the plurality of log lines.
7 . The system of claim 6 , wherein:
the at least one processor is configured to generate the first grouping of log lines based on syntactic and semantic similarity between the log lines within the first grouping of log lines.
8 . The system of claim 1 , wherein:
the at least one processor is configured to detect that a second plurality of template identifiers should be used to update the positive pairings and the negative pairings; and
the at least one processor is configured to fine-tune the large language model using the updated positive pairings and the updated negative pairings.
9 . The system of claim 1 , wherein:
the at least one processor is configured to utilize a generative model to generate a first natural language description for the first log line of the plurality of log lines.
10 . The system of claim 1 , wherein:
the large language model comprises a security specific large language model; and
the at least one processor is configured to fine-tune the security specific large language model using the positive pairings and the negative pairings.
11 . The system of claim 1 , wherein:
the at least one processor is configured to determine the plurality of template identifiers based on syntactic and semantic similarity between the natural language descriptions of the plurality of natural language descriptions; and
the at least one processor is configured to detect that the response identifies a denial-of-service attack and the security risk mitigation action comprises blocking IP traffic from sources identified in the response.
12 . A method, comprising:
receiving security data, the security data includes a plurality of log lines;
mapping a first log line of the plurality of log lines to an event identifier associated with a type of security event;
setting a window size for the first log line of the plurality of log lines based on the event identifier for the first log line;
partitioning the first log line based on the window size;
determining a prompt for generating a plurality of natural language descriptions corresponding with the plurality of log lines;
generating, using the prompt, the plurality of natural language descriptions corresponding with the plurality of log lines;
determining a plurality of template identifiers based on similarity between natural language descriptions of the plurality of natural language descriptions;
generating, using the plurality of template identifiers, positive pairings and negative pairings of the plurality of natural language descriptions;
fine-tuning a large language model using the positive pairings and the negative pairings, the fine-tuning the large language model includes fine-tuning the large language model such that the positive pairings are given similar embeddings with at most a first embedding distance while the negative pairings are given different embeddings with at least a second embedding distance greater than the first embedding distance;
storing the large language model;
generating, using the large language model, a response that identifies a cyber attack; and
performing a security risk mitigation action based on the response.
13 . The method of claim 12 , further comprising:
detecting that a second plurality of template identifiers should be used to update the positive pairings and the negative pairings;
updating, using the second plurality of template identifiers, the positive pairings and the negative pairings; and
fine-tuning the large language model using the updated positive pairings and the updated negative pairings.
14 . The method of claim 12 , wherein:
the first embedding distance corresponds with a cosine similarity distance;
the large language model is stored using a data storage device; and
the security risk mitigation action comprises blocking IP traffic from sources identified in the response.