Hybrid scanning using deferral learning
Systems and methods for performing malware scanning for a service provider network are disclosed. In response to accessing one or more files, file attributes may be determined. Confidence values may be generated based on the file attributes and may be used to select a scan operation. Such scan operations may include a scan operation using a 3 rd party malware scan algorithm or a scan operation using a machine learning generated model. The selection of the scan operation may be performed based on a deferral learning model.
1 . A system, comprising:
one or more hardware computing devices configured to implement a scanner, wherein the one or more hardware computing devices are configured to:
access one or more files to be scanned for malware;
determine one or more file attributes associated with the one or more files; and
select a scan operation to use to scan the one or more files for malware, based on the determined one or more file attributes, from among a plurality of scan operations comprising:
a) a first scan operation using a third-party scan algorithm for detecting malware; and
b) a second scan operation using a machine learning generated model for detecting malware,
wherein to perform the selection based on the determined one or more file attributes, the one or more hardware computing devices are configured to:
use a deferral learning model to select the scan operation to use, wherein the deferral learning model has been trained to select the scan operation based on learned confidence values associated with the one or more file attributes, wherein the learned confidence values are based on accuracy of malware detection in previous results of the first scan operation for other files having the one or more file attributes or previous results of the second scan operation for the other files having the one or more file attributes.
2 . The system of claim 1 , wherein the one or more hardware computing devices are configured to:
receive training data that has been labeled to indicate malware comprised in the training data;
train a machine learning model for the second scan operation using the training data; and
train the deferral learning model using the training data.
3 . The system of claim 2 , wherein the one or more hardware computing devices are configured to:
generate the learned confidence values based on whether predicted malware detection results of the first or second scan operation match malware detection results indicated in labels in the training data.
4 . The system of claim 1 , wherein the one or more hardware computing devices are configured to:
receive user selection preferences towards false positives or false negatives; and
generate the learned confidence values based, at least in part, on the user selection preferences.
5 . The system of claim 1 , wherein the selection of the scan operation is by default biased towards the first scan operation.
6 . The system of claim 1 , wherein the one or more hardware computing devices are configured to:
provide a file with learned confidence values lower than a threshold to a service, wherein the service analyzes the file based on human action.
7 . One or more non-transitory computer-readable storage media storing program instructions that, when executed on or across one or more processors, implement a scanner and cause the scanner to:
determine one or more file attributes associated with one or more files; and
select a scan operation to use to scan the one or more files, based on the determined one or more file attributes, from among a plurality of scan operations comprising:
a) a first scan operation using a third-party scan algorithm for detecting malware; and
b) a second scan operation using a machine learning generated model for detecting malware,
wherein to perform the selection based on the determined one or more file attributes, the program instructions, when executed on or across the one or more processors, further cause the one or more processors to:
use a deferral learning model to select the scan operation to use, wherein the deferral learning model has been trained to select the scan operation based on learned confidence values associated with the one or more file attributes, wherein the learned confidence values are based on accuracy of malware detection in previous results of the first scan operation for other files having the one or more file attributes or previous results of the second scan operation for the other files having the one or more file attributes.
8 . The one or more non-transitory computer readable storage media of claim 7 , wherein the instructions, when executed on or across the one or more processors, further cause the one or more processors to:
receive training data that has been labeled to indicate malware comprised in the training data;
train a machine learning model for the second scan operation using the training data; and
train the deferral learning model using the training data.
9 . The one or more non-transitory computer readable storage media of claim 8 , wherein the instructions, when executed on or across the one or more processors, further cause the one or more processors to:
generate the learned confidence values based on whether predicted malware detection results of the first or second scan operation match malware detection results indicated in labels in the training data.
10 . The one or more non-transitory computer readable storage media of claim 7 , wherein the instructions, when executed on or across the one or more processors, further cause the one or more processors to:
receive user selection preferences towards false positives or false negatives; and
generate the learned confidence values based, at least in part, on the user selection preferences.
11 . The one or more non-transitory computer readable storage media of claim 7 , wherein the instructions, when executed on or across the one or more processors, further cause the one or more processors to:
generate a response action to a user based, at least in part on a malware detection result of the selected first or second scan operation.
12 . The one or more non-transitory computer readable storage media of claim 11 , wherein the response action comprises a malware detection notification.
13 . The one or more non-transitory computer readable storage media of claim 11 , wherein the response action causes the file with the malware to be removed.
14 . The one or more non-transitory computer readable storage media of claim 11 , wherein the response action causes a source of the malware to be blocked.
15 . A method, comprising:
determining one or more file attributes associated with one or more files; and
selecting a scan operation to use to scan the one or more files, based on the determined one or more attributes, from among a plurality of scan operations comprising:
a) a first scan operation using a third-party scan algorithm for detecting malware; and
b) a second scan operation using a machine learning generated model for detecting malware,
wherein performing the selection based on the determined one or more file attributes comprises:
using a deferral learning model to select the scan operation to use, wherein the deferral learning model has been trained to select the scan operation based on learned confidence values associated with the one or more file attributes, wherein the learned confidence values are based on accuracy of malware detection in previous results of the first scan operation for other files having the one or more file attributes or previous results of the second scan operation for the other files having the one or more file attributes.
16 . The method of claim 15 , further comprising:
receiving training data that has been labeled to indicate malware comprised in the training data;
training a machine learning model for the second scan operation using the training data; and
training the deferral learning model using the training data.
17 . The method of claim 16 , further comprising:
generating the learned confidence values based on whether predicted malware detection results of the first or second scan operation match malware detection results indicated in labels in the training data.
18 . The method of claim 15 , further comprising:
receiving user selection preferences towards false positives or false negatives; and
generating the learned confidence values based, at least in part, on the user selection preferences.
19 . The method of claim 15 , wherein the one or more files comprise one or more Linux-based files.
20 . The method of claim 15 , wherein the first and second scan operations are configured to detect malware including viruses, ransomware, cryptominers, worms, viruses, trojans, bot[net]s, adware, spyware, or rootkits.