Adaptive and recursive filtering for sample submission
View Patent ↗Disclosed are systems, apparatuses, computer readable media, and methods of using an adaptive (i.e., learning) client-server system to enhance the efficiency of sample submissions, e.g., the submission of samples of malware programs to a server dictionary. The server in such a system may accumulate a dictionary of known programs and/or portions of programs that have been submitted by many different client devices over time, representing all the various programs that they have encountered. If a portion of a particular file submission is already “available” to the server (i.e., it is a portion that the server has already analyzed and stored), it will be excluded from all future sample transmissions. The server will gradually accumulate a rich dictionary of common program portions, thus requesting transmission only of previously unseen portions. Such systems, apparatuses, computer readable media, and methods may therefore be used to reduce transmission times needed for future sample submissions.
1. A non-transitory computer readable medium comprising computer executable instructions stored thereon to cause one or more processing units to:
identify a sample for submission to a server, wherein the server is configured to perform malware analysis on the submitted sample;
receive an indication from the server whether the sample is available to the server; and
if the indication is that the sample is not available to the server:
split the sample into a plurality of portions according to a set of rules;
filter the plurality of portions, wherein the instructions to filter comprise instructions to discard at least a part of the respective portions;
create hashes of each of the plurality of filtered portions, wherein each respective filtered portion comprises parts of the portion that were not discarded by the filtering, and wherein each of the created hashes comprises a fingerprint;
submit the fingerprints of each of the plurality of filtered portions, along with their respective locations relative to a beginning of the sample, to the server;
receive an indication from the server whether each of the plurality of portions is available to the server; and
submit to the server only those filtered portions wherein the server indicated that the portion is not available to the server,
wherein each submitted filtered portion remains sufficiently similar to the respective portion before filtering, allowing the server to perform the malware analysis.
2. The non-transitory computer readable medium of claim 1 , wherein at least one of the submitted filtered portions has had at least one of the following parts discarded: a block of zeroes; spaces; data-fill blocks; repeated patterns; inactive code branches; and dead code branches.
3. The non-transitory computer readable medium of claim 1 , wherein, if the indication is that the sample is available to the server, no portion of the sample is submitted to the server.
4. The non-transitory computer readable medium of claim 1 , wherein the set of rules comprises a set of rules that has been pre-shared from the server.
5. The non-transitory computer readable medium of claim 1 , wherein at least one of the plurality of portions comprises a library file.
6. The non-transitory computer readable medium of claim 1 , wherein, if the indication is that one of the portions of the plurality of portions is not available to the server, the instructions further cause the one or more processing units to recursively split the one portion into a plurality of sub-portions according to the set of rules.
7. The non-transitory computer readable medium of claim 6 , wherein the instructions further cause the one or more processing units to continue recursively splitting portions into a plurality of sub-portions according to the set of rules until either: receiving an indication from the server that a sub-portion is available to the server; or the set of rules determines that no further splitting is required.
8. The non-transitory computer readable medium of claim 1 , wherein the instructions to identify a sample for submission to a server comprise instructions to identify a sample for malware analysis.
9. A non-transitory computer readable medium comprising computer executable instructions stored thereon to cause one or more processing units to:
send a set of sample splitting rules to a client;
receive a hash of a sample from the client;
determine whether the sample is available to a dictionary communicatively coupled to the one or more processing units;
provide an indication to the client whether the sample is available; and
if the indication is that the sample is not available:
receive, from the client, a plurality of hashes of filtered portions of the sample, along with their respective locations relative to a beginning of the sample,
wherein the filtered portions of the sample have been created by splitting the sample according to the set of rules,
wherein the filtered portions of the sample comprise portions of the sample that remain after at least a part of the portion of the sample has been discarded,
wherein each filtered portion remains sufficiently similar to the respective portion before filtering, allowing the malware analysis to be performed, and
wherein each of the plurality of hashes of filtered portions comprises a fingerprint;
determine whether each of a plurality of portions corresponding to the received plurality of hashes of filtered portions is available to the dictionary;
provide an indication to the client whether each of the plurality of portions is available to the dictionary;
receive, from the client, filtered versions of only those portions wherein the portion is not available to the dictionary;
store the received filtered portions in the dictionary; and
perform malware analysis on the sample.
10. The non-transitory computer readable medium of claim 9 , further comprising instructions to cause the one or more processing units to reconstruct the sample by merging the received filtered portions and the portions already available to the dictionary.
11. The non-transitory computer readable medium of claim 9 , wherein at least one of the received filtered portions has had at least one of the following parts discarded: a block of zeroes; spaces; data-fill blocks; repeated patterns; inactive code branches; and dead code branches.
12. The non-transitory computer readable medium of claim 9 , further comprising instructions to cause the one or more processing units to replace at least one of the received filtered portions with an unfiltered version of the portion.
13. The non-transitory computer readable medium of claim 9 , further comprising instructions stored thereon to cause one or more processing units to:
analyze the received filtered portions;
determine an updated set of sample splitting rules based, at least in part, on the analysis of the received portions; and
send the updated set of sample splitting rules to the client.
14. A method of sample analysis, comprising:
sending a set of sample splitting rules from a server to a client;
receiving at the server a hash of a sample from the client;
determining at the server whether the sample is available to a dictionary communicatively coupled to the server;
providing by the server an indication to the client whether the sample is available to the server; and
if the indication is that the sample is not available:
receiving, at the server, a plurality of hashes of filtered portions of the sample, along with their respective locations relative to a beginning of the sample,
wherein the filtered portions of the sample have been created by splitting the sample according to the set of rules,
wherein the filtered portions comprise portions of the sample that remain after at least a part of the portion of the sample has been discarded,
wherein each filtered portion remains sufficiently similar to the respective portion before filtering, allowing the server to perform the malware analysis, and
wherein each of the plurality of hashes of filtered portions comprises a fingerprint;
determining at the server whether each of a plurality of portions corresponding to the received plurality of hashes of filtered portions is available to the dictionary;
providing, by the server, an indication to the client whether each of the plurality of portions is available to the dictionary;
receiving, at the server, filtered versions of only those portions wherein the portion is not available to the dictionary;
storing the received filtered portions in the dictionary; and
performing malware analysis on the sample.
15. The method of claim 14 , further comprising the act of reconstructing the sample by merging the received portions and the portions already available to the dictionary at the server.
16. The method of claim 14 , wherein the act of performing malware analysis on the sample is performed at the server.
17. The method of claim 14 , wherein at least one of the received filtered portions has had at least one of the following parts discarded: a block of zeroes; spaces; data-fill blocks; repeated patterns; inactive code branches; and dead code branches.
18. The method of claim 14 , further comprising the act of replacing at the server at least one of the received filtered portions with an unfiltered version of the portion.
19. The method of claim 14 , further comprising the acts of:
analyzing the received portions at the server;
determining an updated set of sample splitting rules based, at least in part, on the analysis of the received portions; and
sending the updated set of sample splitting rules to the client.
20. An apparatus configured to perform sample analysis, comprising:
a memory; and
one or more processing units, communicatively coupled to the memory, wherein the memory stores instructions to configure the one or more processing units to:
identify a sample for submission to a server, wherein the server is configured to perform malware analysis on the submitted sample;
filter the sample, wherein the instructions to filter comprise instructions to discard at least a part of the sample;
create a hash of the filtered sample;
submit the hash of the filtered sample to the server;
receive an indication from the server whether the filtered sample is available to the server; and
if the indication is that the filtered sample is not available to the server:
split the sample into a plurality of portions according to a set of rules;
filter the plurality of portions, wherein the instructions to filter the plurality of portions comprise instructions to discard at least a part of the respective portions;
create hashes of each of the filtered plurality of portions, wherein each respective filtered portion comprises parts of the portion that were not discarded by the filtering, and wherein each of the created hashes comprises a fingerprint;
submit the fingerprints of each of the filtered plurality of portions, along with their respective locations relative to a beginning of the sample, to the server;
receive an indication from the server whether each of the plurality of portions is available to the server; and
submit to the server only those portions wherein the server indicated that the portion is not available to the server,
wherein each submitted filtered portion remains sufficiently similar to the respective portion before filtering, allowing the server to perform the malware analysis.
21. The apparatus of claim 20 , wherein, if the indication is that one of the portions of the plurality of portions is not available to the server, the instructions further cause the one or more processing units to recursively split the one portion into a plurality of sub-portions according to the set of rules.
22. The apparatus of claim 20 , wherein the instructions to identify a sample for submission to a server comprise instructions to identify a sample for malware analysis.