Applications of machine learning models to a binary search engine based on an inverted index of byte sequences
Techniques for searching an inverted index associating byte sequences of a fixed length and files that contain those byte sequences are described herein. Byte sequences comprising a search query are determined and searched in the inverted index. In some examples, training data for training machine learning model(s) may be created using pre-featured data from the inverted index. In various examples, training data may be used to retrain a ML model until the ML model meets a criterion. In some examples, the trained ML model may be used to perform searches on the inverted index and classify files.
1. A method comprising:
generating a first machine learning (ML) model using pre-featured training data from an inverted index, the first ML model including a string section and a first condition section, the string section including one or more features, the first condition section including one or more weights corresponding to the one or more features;
determining a first decision value of the first ML model based at least in part on predetermined reference data including malicious files;
generating a second ML model using the pre-featured training data and false positives from determining the first decision value, the second ML model including the string section and a second condition section, the second condition section including at least one different weight of the one or more weights;
determining a second decision value of the second ML model based at least in part on the predetermined reference data;
performing a first search of the inverted index using the first ML model; and
performing a second search of the inverted index using the second ML model.
2. The method of claim 1 , further comprising:
determining that the first decision value is below a first confidence threshold;
determining that the second decision value meets the first confidence threshold; and
associating the second ML model with a first confidence level based at least in part on the second decision value meeting the first confidence threshold, the first confidence level including a first confidence interval.
3. The method of claim 2 , further comprising:
determining that the second decision value is below a second confidence threshold associated with a second confidence level, the second confidence level including a second confidence interval; and
generating a third ML model using the pre-featured training data and false positives from determining the second decision value.
4. The method of claim 3 , further comprising:
determining that a third decision value associated with the third ML model meets the second confidence threshold; and
associating the third ML model with the second confidence level.
5. The method of claim 1 , further comprising expressing the second ML model as a query language model to perform search of the inverted index.
6. A computer implemented method comprising:
generating an inverted index that specifies byte sequences from a corpus of files;
creating training data using pre-featured data from the inverted index, the pre-featured data including features in a string portion of the pre-featured data;
generating a first machine learning (ML) model using the training data wherein the first ML model includes a first feature associated with a first weight and a second feature associated with a second weight;
determining a decision value of the first ML model is outside of a confidence interval, wherein the decision value is based at least in part on a ratio of false positives to total searches;
generating a second ML model using the training data and the false positives from determining the decision value; and
initiating a search query of the inverted index at least partly in response to addition of one or more new files to the inverted index, wherein the inverted index specifies byte sequences of a fixed length.
7. The method of claim 6 , further comprising:
applying the first ML model to search a test sample of the inverted index; and
determining that the decision value for the first ML model is within the confidence interval.
8. The method of claim 7 , further comprising:
applying the first ML model to search the inverted index.
9. The method of claim 8 , further comprising:
validating search results with the first ML model; and
returning the search results.
10. The method of claim 7 , further comprising:
expressing the first ML model as a query language to automatically perform a search query on the inverted index.
11. The method of claim 7 , further comprising:
determining a second decision value is outside of a second confidence interval based at least in part on a second ratio of second false positives to second total searches.
12. The method of claim 11 , further comprising:
generate a third ML model using the training data and the second false positives.
13. The method of claim 12 , wherein generating the third ML model includes repeating training for the third ML model until the second decision value is within the second confidence interval.
14. The method of claim 13 , wherein repeating the training includes adjusting at least one of the first weight or the second weight.
15. The method of claim 6 , wherein the corpus of files includes a corpus of malware files.
16. One or more non-transitory computer readable media storing computer-executable instructions that, when executed, cause one or more processors to perform operations comprising:
generating an inverted index that specifies byte sequences from a corpus of files;
creating training data using pre-featured data from the inverted index, the pre-featured data including features in a string portion of the pre-featured data;
generating a first machine learning (ML) model using the training data, wherein the first ML model includes a first feature associated with a first weight and a second feature associated with a second weight;
determining a decision value of the first ML model is outside of a confidence interval, wherein the decision value is based at least in part on a ratio of false positives to total searches;
generating a second ML model using the training data and the false positives from determining the decision value; and
initiating a search query of the inverted index at least partly in response to addition of one or more new files to the inverted index, wherein the inverted index specifies byte sequences of a fixed length.
17. The one or more non-transitory computer readable media of claim 16 , the operations further comprising:
applying the first ML model to search a test sample of the inverted index; and
determining that a decision value for the first ML model is within a confidence interval.
18. The one or more non-transitory computer readable media of claim 17 , the operations further comprising:
applying the first ML model to search the inverted index;
validating search results with the first ML model; and
returning the search results.
19. The one or more non-transitory computer readable media of claim 17 , the operations further comprising:
determining that the decision value is outside of a second confidence interval; and
repeating the training for the second ML model until the decision value meets is within the second confidence interval.
20. The one or more non-transitory computer readable media of claim 16 , the operations further comprising:
determining a plurality of byte sequences of the fixed length, the plurality of byte sequences corresponding to a set of search strings of the first ML model.