Binary search of byte sequences using inverted indices
Techniques for searching an inverted index associating byte sequences of a fixed length and files that contain those byte sequences are described herein. Byte sequences comprising a search query are determined and searched in the inverted index, and an intersection of the results is determined and returned as a response to the search query. Further, search queries in the form of expressions including search terms and logical operators are searched in the inverted index and evaluated using a syntax tree constructed based on the logical operators. Also, byte sequences comprising a file are searched in the inverted index and results of the search are used to generate signatures and fuzzy hashes.
1. A computing system comprising:
one or more processors; and
memory storing computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
receiving a search query based on user input, wherein the search query is an expression comprising a plurality of search terms and at least one logical operator;
determining one or more target byte sequences of a fixed length that correspond to the search query;
searching for the one or more target byte sequences in an inverted index, wherein the inverted index maps a plurality of byte sequences of the fixed length to one or more file identifiers of one or more files that include individual ones of the plurality of byte sequences;
generating a syntax tree based on the at least one logical operator of the search query; and
identifying, by evaluating search results of the searching using the syntax tree, at least one file that includes the one or more target byte sequences.
2. The computing system of claim 1 , wherein the operations further comprise generating the inverted index from a corpus of files that contains the one or more files.
3. The computing system of claim 1 , wherein storage of the inverted index is distributed across a plurality of computing devices.
4. The computing system of claim 1 , wherein the one or more target byte sequences are n-grams and the fixed length is four bytes.
5. The computing system of claim 1 , wherein the one or more target byte sequences are determined based on individual search terms, of the plurality of search terms.
6. The computing system of claim 1 , wherein a plurality of leaves of the syntax tree are associated with the plurality of search terms.
7. The computing system of claim 6 , wherein the operations further comprise validating that the at least one file satisfies the expression.
8. The computing system of claim 2 , wherein the corpus of files comprises at least one of:
malware files,
binary files,
executable files, or
unstructured files.
9. A computer-implemented method comprising:
receiving a search query based on user input, wherein the search query is an expression comprising a plurality of search terms and at least one logical operator;
determining one or more target byte sequences of a fixed length that correspond to the search query;
searching for the one or more target byte sequences in an inverted index, wherein the inverted index maps a plurality of byte sequences of the fixed length to one or more file identifiers of one or more files that include individual ones of the plurality of byte sequences;
generating a syntax tree based on the at least one logical operator of the search query; and
identifying, by evaluating search results of the searching using the syntax tree, at least one file that includes the one or more target byte sequences.
10. The computer-implemented method of claim 9 , further comprising generating the inverted index from a corpus of files that contains the one or more files.
11. The computer-implemented method of claim 9 , wherein the one or more target byte sequences are n-grams and the fixed length is four bytes.
12. The computer-implemented method of claim 9 , wherein the one or more target byte sequences are determined based on individual search terms, of the plurality of search terms.
13. The computer-implemented method of claim 9 , wherein a plurality of leaves of the syntax tree are associated with the plurality of search terms.
14. The computer-implemented method of claim 13 , further comprising validating that the at least one file satisfies the expression.
15. One or more non-transitory computer-readable media storing computer-executable instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:
receiving a search query based on user input, wherein the search query is an expression comprising a plurality of search terms and at least one logical operator;
determining one or more target byte sequences of a fixed length that correspond to the search query;
searching for the one or more target byte sequences in an inverted index, wherein the inverted index maps a plurality of byte sequences of the fixed length to one or more file identifiers of one or more files that include individual ones of the plurality of byte sequences
generating a syntax tree based on the at least one logical operator of the search query; and
identifying, by evaluating search results of the searching using the syntax tree, at least one file that includes the one or more target byte sequences.
16. The one or more non-transitory computer-readable media of claim 15 , wherein the operations further comprise generating the inverted index from a corpus of files that contains the one or more files.
17. The one or more non-transitory computer-readable media of claim 15 , wherein the one or more target byte sequences are determined based on individual search terms, of the plurality of search terms.
18. The one or more non-transitory computer-readable media of claim 15 , wherein the one or more target byte sequences are n-grams and the fixed length is four bytes.
19. The one or more non-transitory computer-readable media of claim 15 , wherein a plurality of leaves of the syntax tree are associated with the plurality of search terms.
20. The one or more non-transitory computer-readable media of claim 19 , wherein the operations further comprise validating that the at least one file satisfies the expression.