Generating security language queries
A computer-implemented method of generating a security language query from a user input query includes receiving, at a computer system, an input security hunting user query indicating a user intention; selecting, using a trained machine learning model and based on the input security hunting query, an example user security hunting query and corresponding example security language query; generating, using the trained machine learning model, query metadata from the input security hunting query; generating a prompt, the prompt comprising: the input security hunting user query; the selected example user security hunting query and the corresponding example security language query; and the generated query metadata; inputting the prompt to a large language model; receiving a security language query from the large language model corresponding to the input security hunting query reflective of the user intention.
1. A computer-implemented method comprising:
receiving a training data set comprising a plurality of user security hunting queries, corresponding ground truth security language queries, and corresponding query metadata;
generating a plurality of probe prompts from the training data set, wherein generating each probe prompt comprises:
selecting one of the plurality of user security hunting queries as a subject of the probe prompt, and
randomly selecting a subset of the plurality of user security hunting queries and corresponding ground truth security language queries as examples for inclusion into the probe prompt;
inputting the probe prompts to a large language model;
receiving, as output from the large language model, security language queries corresponding to the subjects of the probe prompts;
comparing the security language queries for the probe prompts to the corresponding ground truth security language queries;
calculating probe scores for the probe prompts based on the comparison; and
training a machine learning model based on the calculated probe scores and the training data set, the machine learning model being trained to:
receive an input user security hunting user query,
generate output scores reflective of a utility of the user security hunting queries and corresponding ground truth security language queries of the training data set to the input user security hunting user query, and
select a user security hunting query and corresponding ground truth security language query of the training data set as an example for inclusion into a prompt for input to the large language model based on the output scores, and
generate query metadata for inclusion into the prompt for input to the large language model from the input security hunting user query.
2. The method of claim 1 , wherein the trained model comprises:
a base large language model;
a pooling layer;
a ranking head; and
a classification head,
wherein the ranking head is trained to select the user security hunting query and corresponding ground truth security language query, and the classification head is trained to generate the query metadata.
3. The method of claim 1 , comprising generating the training data set by:
receiving an initial training data set comprising a plurality of user security hunting queries, corresponding ground truth security language queries, and corresponding query metadata;
receiving a security query set comprising security language queries and corresponding textual descriptions of the security language queries;
generating a prompt including:
a user security hunting query and corresponding ground truth security language query drawn from the initial training data set, and
a security language query and corresponding textual description from the security query set;
inputting the prompt to a natural language generation large language model;
receiving, as output from the natural language generation large language model, a user security hunting query corresponding to the security language query from the security query set,
including the received user security hunting query and the security language query from the security query set in the training data set.
4. The method of claim 1 , comprising:
backtranslating the user security hunting queries to generate backtranslated security hunting queries; and
including the backtranslated security hunting queries in the training data set.
5. The method of claim 1 , comprising:
fitting a linear model to the probe scores;
generating, from the linear model, rankings indicative of usefulness of the examples included in the probe prompts to the subjects of the probe prompts;
wherein training the machine learning model to generate the output scores to select a user security hunting query and corresponding ground truth security language query of the training data set is based on the ranking.
6. The method of claim 5 , wherein training the machine learning model based on the ranking comprises:
selecting a pair of examples from the ranking;
inputting the pair of examples to the machine learning model;
training the machine learning model to minimize a pairwise cross-entropy loss calculated from the pair of examples.
7. The method of claim 1 , comprising:
generating a plurality of hyperparameter probe prompts, wherein generating each hyperparameter probe prompt comprises:
selecting one of the plurality of user security hunting queries as a subject of the hyperparameter probe prompt, and
randomly selecting values for a hyperparameter of the machine learning model;
inputting the generated hyperparameter probe prompts to the large language model;
receiving, as output from the large language model, security language queries corresponding to the subjects of the hyperparameter probe prompt;
comparing the received security language queries for the hyperparameter probe prompts to the corresponding ground truth security language queries;
calculating hyperparameter probe scores for the hyperparameter probe prompts based on the comparison; and
training the machine learning model to select a value for the hyperparameter based on the calculated hyperparameter probe scores.
8. A system comprising a processor and a storage, the storage storing computer-readable instructions which, when executed by the processor, cause the system to perform operations comprising:
receiving a training data set comprising a plurality of user security hunting queries, corresponding ground truth security language queries, and corresponding query metadata;
generating a plurality of probe prompts from the training data set, wherein generating each probe prompt comprises:
selecting one of the plurality of user security hunting queries as a subject of the probe prompt, and
randomly selecting a subset of the plurality of user security hunting queries and corresponding ground truth security language queries as examples for inclusion into the probe prompt;
inputting the probe prompts to a large language model;
receiving, as output from the large language model, security language queries corresponding to the subjects of the probe prompts;
comparing the security language queries for the probe prompts to the corresponding ground truth security language queries;
calculating probe scores for the probe prompts based on the comparison; and
training a machine learning model based on the calculated probe scores and the training data set, the machine learning model being trained to:
receive an input user security hunting user query,
generate output scores reflective of a utility of the user security hunting queries and corresponding ground truth security language queries of the training data set to the input user security hunting user query, and
select a user security hunting query and corresponding ground truth security language query of the training data set as an example for inclusion into a prompt for input to the large language model based on the output scores, and generate query metadata for inclusion into the prompt for input to the large language model from the input security hunting user query.
9. The system of claim 8 , wherein the trained model comprises:
a base large language model;
a pooling layer;
a ranking head; and
a classification head,
wherein the ranking head is trained to select the user security hunting query and corresponding ground truth security language query, and the classification head is trained to generate the query metadata.
10. The system of claim 8 , the operations further comprising generating the training data set by:
receiving an initial training data set comprising a plurality of user security hunting queries, corresponding ground truth security language queries, and corresponding query metadata;
receiving a security query set comprising security language queries and corresponding textual descriptions of the security language queries;
generating a prompt including:
a user security hunting query and corresponding ground truth security language query drawn from the initial training data set, and
a security language query and corresponding textual description from the security query set;
inputting the prompt to a natural language generation large language model;
receiving, as output from the natural language generation large language model, a user security hunting query corresponding to the security language query from the security query set,
including the received user security hunting query and the security language query from the security query set in the training data set.
11. The system of claim 8 , the operations further comprising:
backtranslating the user security hunting queries to generate backtranslated security hunting queries; and
including the backtranslated security hunting queries in the training data set.
12. The system of claim 8 , the operations further comprising:
fitting a linear model to the probe scores;
generating, from the linear model, rankings indicative of usefulness of the examples included in the probe prompts to the subjects of the probe prompts;
wherein training the machine learning model to generate the output scores to select a user security hunting query and corresponding ground truth security language query of the training data set is based on the ranking.
13. The system of claim 12 , wherein training the machine learning model based on the ranking comprises:
selecting a pair of examples from the ranking;
inputting the pair of examples to the machine learning model;
training the machine learning model to minimize a pairwise cross-entropy loss calculated from the pair of examples.
14. The system of claim 8 , the operations further comprising:
generating a plurality of hyperparameter probe prompts, wherein generating each hyperparameter probe prompt comprises:
selecting one of the plurality of user security hunting queries as a subject of the hyperparameter probe prompt, and
randomly selecting values for a hyperparameter of the machine learning model;
inputting the generated hyperparameter probe prompts to the large language model;
receiving, as output from the large language model, security language queries corresponding to the subjects of the hyperparameter probe prompt;
comparing the received security language queries for the hyperparameter probe prompts to the corresponding ground truth security language queries;
calculating hyperparameter probe scores for the hyperparameter probe prompts based on the comparison; and
training the machine learning model to select a value for the hyperparameter based on the calculated hyperparameter probe scores.
15. A non-transitory computer-readable storage medium storing computer-readable instructions which, when executed by a processor, cause the processor to perform operations comprising:
receiving a training data set comprising a plurality of usersecurity hunting queries, corresponding ground truth security language queries, and corresponding query metadata;
generating a plurality of probe prompts from the training data set, wherein generating each probe prompt comprises:
selecting one of the plurality of user security hunting queries as a subject of the probe prompt, and
randomly selecting a subset of the plurality of user security hunting queries and corresponding ground truth security language queries as examples for inclusion into the probe prompt;
inputting the probe prompts to a large language model;
receiving, as output from the large language model, security language queries corresponding to the subjects of the probe prompts;
comparing the security language queries for the probe prompts to the corresponding ground truth security language queries;
calculating probe scores for the probe prompts based on the comparison; and
training a machine learning model based on the calculated probe scores and the training data set, the machine learning model being trained to:
receive an input user security hunting user query,
generate output scores reflective of a utility of the user security hunting queries and corresponding ground truth security language queries of the training data set to the input usersecurity hunting user query, and
select a user security hunting query and corresponding ground truth security language query of the training data set as an example for inclusion into a prompt for input to the large language model based on the output scores, and generate query metadata for inclusion into the prompt for input to the large language model from the input security hunting user query.
16. The non-transitory computer-readable storage medium of claim 15 , wherein the trained model comprises:
a base large language model;
a pooling layer;
a ranking head; and
a classification head,
wherein the ranking head is trained to select the user security hunting query and corresponding ground truth security language query, and the classification head is trained to generate the query metadata.
17. The non-transitory computer-readable storage medium of claim 15 , the operations further comprising generating the training data set by:
receiving an initial training data set comprising a plurality of user security hunting queries, corresponding ground truth security language queries, and corresponding query metadata;
receiving a security query set comprising security language queries and corresponding textual descriptions of the security language queries;
generating a prompt including:
a user security hunting query and corresponding ground truth security language query drawn from the initial training data set, and
a security language query and corresponding textual description from the security query set;
inputting the prompt to a natural language generation large language model;
receiving, as output from the natural language generation large language model, a user security hunting query corresponding to the security language query from the security query set,
including the received user security hunting query and the security language query from the security query set in the training data set.
18. The non-transitory computer-readable storage medium of claim 15 , the operations further comprising:
backtranslating the user security hunting queries to generate backtranslated security hunting queries; and
including the backtranslated security hunting queries in the training data set.
19. The non-transitory computer-readable storage medium of claim 15 , the operations further comprising:
fitting a linear model to the probe scores;
generating, from the linear model, rankings indicative of usefulness of the examples included in the probe prompts to the subjects of the probe prompts;
wherein training the machine learning model to generate the output scores to select a user security hunting query and corresponding ground truth security language query of the training data set is based on the ranking.
20. The non-transitory computer-readable storage medium of claim 19 , wherein training the machine learning model based on the ranking comprises:
selecting a pair of examples from the ranking;
inputting the pair of examples to the machine learning model;
training the machine learning model to minimize a pairwise cross-entropy loss calculated from the pair of examples.