Deriving statistically probable and statistically relevant indicator of compromise signature for matching engines
Methods and systems are provided for a histogram model configuring a computing system to derive an indicator of compromise signature based on a sliding window index of identified malware samples, and a matching rule constructor configuring a computing system to generate matching signatures by selecting statistically relevant n-grams of an unidentified file sample. A matching rule constructor configures the computing system to construct a matching rule including, as a signature, 32 n-grams found in the unidentified file sample which occur most frequently, and another 32 n-grams found in the unidentified file sample which occur least frequently amongst records of the threat database across 32 discrete file size ranges. These functions can configure backend operations to a sample identification operation performed by a user operating a client computing device, in a fashion that does not require a user to manually discern strings from the unidentified file sample to derive a signature for the matching engine to search against the threat database.
1. A method comprising:
searching a threat database using a matching rule derived from an n-gram from an unidentified file sample, returning a set of matched records;
outputting a histogram corresponding to the n-gram, the histogram comprising a plurality of bins each counting, for a respective file size range of a plurality of file size ranges, matched records of the threat database in that respective file size range;
aggregating a plurality of histograms corresponding to respective n-grams into an aggregated histogram; and
constructing a matching rule according to at least most frequent n-grams of each bin of the aggregated histogram.
2. The method of claim 1 , wherein the n-gram is taken from the unidentified file sample according to parameter values for a value of n and a size of an interval.
3. The method of claim 1 , wherein the threat database comprises identified file samples indexed by a sliding window index, the sliding window index comprising n-grams taken at intervals of bytes.
4. The method of claim 1 , wherein the bins discretize identified file samples of matched records over exclusive subranges of file sizes.
5. The method of claim 1 , wherein the matching rule comprises a signature, the signature comprising a plurality of n-grams of the unidentified file sample which occur most frequently amongst records of the threat database across a plurality of discrete file size ranges.
6. The method of claim 1 , further comprising constructing the matching rule according to least frequent n-grams of each bin of the aggregated histogram.
7. The method of claim 6 , wherein the matching rule comprises a signature, the signature comprising a plurality of n-grams of the unidentified file sample which occur least frequently amongst records of the threat database across a plurality of discrete file size ranges.
8. A system comprising:
one or more processors; and
memory communicatively coupled to the one or more processors, the memory storing computer-executable modules executable by the one or more processors that, when executed by the one or more processors, perform associated operations, the computer-executable modules comprising:
a matching engine module executable by the one or more processors to search a threat database using a matching rule derived from an n-gram from an unidentified file sample, returning a set of matched records;
a histogram model module executable by the one or more processors to output a histogram corresponding to the n-gram, the histogram comprising a plurality of bins each counting, for a respective file size range of a plurality of file size ranges, matched records of the threat database in that respective file size range;
a histogram aggregator module executable by the one or more processors to aggregate a plurality of histograms corresponding to respective n-grams into an aggregated histogram; and
a matching rule constructor module executable by the one or more processors to construct a matching rule according to at least most frequent n-grams of each bin of the aggregated histogram.
9. The system of claim 8 , wherein the computer-executable modules further comprise a sample tokenizer executable by the one or more processors to take an n-gram from the unidentified file sample according to parameter values for a value of n and a size of an interval.
10. The system of claim 8 , wherein the threat database comprises identified file samples indexed by a sliding window index, the sliding window index comprising n-grams taken at intervals of bytes.
11. The system of claim 8 , wherein the bins discretize identified file samples of matched records over exclusive subranges of file sizes.
12. The system of claim 8 , wherein the matching rule comprises a signature, the signature comprising a plurality of n-grams of the unidentified file sample which occur most frequently amongst records of the threat database across a plurality of discrete file size ranges.
13. The system of claim 8 , wherein the matching rule constructor module is further executable by the one or more processors to construct the matching rule according to least frequent n-grams of each bin of the aggregated histogram.
14. The system of claim 13 , wherein the matching rule comprises a signature, the signature comprising a plurality of n-grams of the unidentified file sample which occur least frequently amongst records of the threat database across a plurality of discrete file size ranges.
15. A non-transitory computer-readable storage medium storing computer-readable instructions executable by one or more processors, that when executed by the one or more processors, cause the one or more processors to perform operations comprising:
searching a threat database using a matching rule derived from an n-gram from an unidentified file sample, returning a set of matched records;
outputting a histogram corresponding to the n-gram, the histogram comprising a plurality of bins each counting, for a respective file size range of a plurality of file size ranges, matched records of the threat database in that respective file size range;
aggregating a plurality of histograms corresponding to respective n-grams into an aggregated histogram; and
constructing a matching rule according to at least most frequent n-grams of each bin of the aggregated histogram.
16. The non-transitory computer-readable storage medium of claim 15 , wherein the n-gram is taken from the unidentified file sample according to parameter values for a value of n and a size of an interval.
17. The non-transitory computer-readable storage medium of claim 15 , wherein the threat database comprises identified file samples indexed by a sliding window index, the sliding window index comprising n-grams taken at intervals of bytes.
18. The non-transitory computer-readable storage medium of claim 15 , wherein the matching rule comprises a signature, the signature comprising a plurality of n-grams of the unidentified file sample which occur most frequently amongst records of the threat database across a plurality of discrete file size ranges.
19. The non-transitory computer-readable storage medium of claim 15 , wherein the operations further comprise constructing the matching rule according to least frequent n-grams of each bin of the aggregated histogram.
20. The non-transitory computer-readable storage medium of claim 19 , wherein the matching rule comprises a signature, the signature comprising a plurality of n-grams of the unidentified file sample which occur least frequently amongst records of the threat database across a plurality of discrete file size ranges.