Binary search of byte sequences using inverted indices
Techniques for searching an inverted index associating byte sequences of a fixed length and files that contain those byte sequences are described herein. Byte sequences comprising a search query are determined and searched in the inverted index, and an intersection of the results is determined and returned as a response to the search query. Further, search queries in the form of expressions including search terms and logical operators are searched in the inverted index and evaluated using a syntax tree constructed based on the logical operators. Also, byte sequences comprising a file are searched in the inverted index and results of the search are used to generate signatures and fuzzy hashes.
1. A computer-implemented method comprising:
for each byte sequence of a fixed length comprising a file, searching an inverted index which specifies byte sequences of the fixed length and, for each specified byte sequence, file identifiers of files that include the specified byte sequence, wherein each file identifier is associated with a security status;
based on results of the search, creating a list of candidate byte sequences, wherein the candidate byte sequences are only found in files with file identifiers that are associated with a malware status or an unknown status;
selecting ones of the candidate byte sequences that are associated with the greatest numbers of file identifiers; and
generating a signature from the selected ones of the candidate byte sequences.
2. The computer-implemented method of claim 1 , wherein the byte sequences of the fixed length are n-grams with a length of four bytes.
3. The computer-implemented method of claim 1 , wherein the searching further comprises determining the byte sequences of the fixed length comprising the file.
4. The computer-implemented method of claim 1 , wherein the security status associated with each file identifier is one of a malware status, a clean status, an unknown status, or another status indicating a level of trust.
5. The computer-implemented method of claim 1 , wherein the searching further comprises filtering out byte sequences known to be found in files with file identifiers associated with a clean status and searching for the remaining byte sequences comprising the file.
6. The computer-implemented method of claim 1 , wherein the creating further comprises determining the security status for each file identifier returned from the search, the security status being metadata for the file identifier.
7. The computer-implemented method of claim 1 , further comprising providing the signature to a security service.
8. A computing device comprising:
one or more processors; and
memory storing computer-executable instructions that, when executed by the one or more processors, perform operations comprising:
for each byte sequence of a fixed length comprising a file, searching an inverted index which specifies byte sequences of the fixed length and, for each specified byte sequence, file identifiers of files that include the specified byte sequence, wherein each file identifier is associated with a security status;
based on results of the search, creating a list of candidate byte sequences, wherein the candidate byte sequences are only found in files with file identifiers that are associated with a malware status or an unknown status;
selecting ones of the candidate byte sequences that are associated with the greatest numbers of file identifiers; and
generating a signature from the selected ones of the candidate byte sequences.
9. The computing device of claim 8 , wherein the byte sequences of the fixed length are n-grams with a length of four bytes.
10. The computing device of claim 8 , wherein the searching further comprises determining the byte sequences of the fixed length comprising the file.
11. The computing device of claim 8 , wherein the security status associated with each file identifier is one of a malware status, a clean status, an unknown status, or another status indicating a level of trust.
12. The computing device of claim 8 , wherein the searching further comprises filtering out byte sequences known to be found in files with file identifiers associated with a clean status and searching for the remaining byte sequences comprising the file.
13. The computing device of claim 8 , wherein the creating further comprises determining the security status for each file identifier returned from the search, the security status being metadata for the file identifier.
14. The computing device of claim 8 , wherein the operations further comprise providing the signature to a security service.
15. One or more non-transitory computer-readable media having stored thereon programming instructions that, when executed by one or more processors, cause the one or more processors to perform operations including:
for each byte sequence of a fixed length comprising a file, searching an inverted index which specifies byte sequences of the fixed length and, for each specified byte sequence, file identifiers of files that include the specified byte sequence, wherein each file identifier is associated with a security status;
based on results of the search, creating a list of candidate byte sequences, wherein the candidate byte sequences are only found in files with file identifiers that are associated with a malware status or an unknown status;
selecting ones of the candidate byte sequences that are associated with the greatest numbers of file identifiers; and
generating a signature from the selected ones of the candidate byte sequences.
16. The one or more non-transitory computer-readable media of claim 15 , wherein the byte sequences of the fixed length are n-grams with a length of four bytes.
17. The one or more non-transitory computer-readable media of claim 15 , wherein the searching further comprises determining the byte sequences of the fixed length comprising the file.
18. The one or more non-transitory computer-readable media of claim 15 , wherein the security status associated with each file identifier is one of a malware status, a clean status, an unknown status, or another status indicating a level of trust.
19. The one or more non-transitory computer-readable media of claim 15 , wherein the searching further comprises filtering out byte sequences known to be found in files with file identifiers associated with a clean status and searching for the remaining byte sequences comprising the file.
20. The one or more non-transitory computer-readable media of claim 15 , wherein the operations further comprise providing the signature to a security service.