Model development and application to identify and halt malware
A method for halting malware includes: monitoring plural file system events with a system driver to detect an occurrence of a file system event having a predetermined file type and log event type; triggering a listening engine for file system event stream data of a file associated with the detection of the file system event, the file system event stream data indicating data manipulation associated with the file due to execution of a process; obtaining one or more feature values for each of plural different feature combinations of plural features of the file based on the file system event stream data; inputting one or more feature values into a data analytics model to predict a target label value based on the one or more feature values of the plural different feature combinations and agnostic to the process; and performing a predetermined operation based on the target label value.
1. An apparatus to detect and halt malware, the apparatus comprising:
memory; and
one or more processors to execute instructions that cause the one or more processors to:
analyze an impact of feature candidates on a target malware prediction, the feature candidates to be associated with a predetermined portion of an original or modified file associated with a file system event, the predetermined portion of the original or modified file including at least one of a first portion of the original file, a first portion of the modified file, a last portion of the original file, or a last portion of the modified file;
determine, using a design of experiments, singular or combinatorial features to be associated with a file system event based on the feature candidates; and
determine a malware prediction based on the features associated with the file system event, the malware prediction associated with a predetermined operation.
2. The apparatus of claim 1 , wherein the instructions, when executed, cause the one or more processors to assign weights to the features associated with the file system event based on the impact of the feature candidates on the target malware prediction.
3. The apparatus of claim 1 , wherein the first portion of the original file is a first 4k bits of the original file, wherein the first portion of the modified file is a first 4k bits of the modified file, wherein the last portion of the original file is a last 4k bits associated with the original file, and wherein the last portion of the modified file is a last 4k bits associated with the modified file.
4. The apparatus of claim 1 , wherein the feature candidates include an entropy of the predetermined portion of the original or modified file, a Monte Carlo pi estimate obtained based on the predetermined portion of the original or modified file, a Monte Carlo pi estimation error based on the predetermined portion of the original or modified file, a serial correlation coefficient associated with the predetermined portion of the original or modified file, a chi square associated with the original or modified file, or an arithmetic mean associated with the original or modified file.
5. The apparatus of claim 1 , wherein the instructions, when executed, cause the one or more processors to:
replace missing or invalid data associated with the file system event;
determine Cartesian products between the feature candidates; and
perform non-linear transformations associated with the feature candidates.
6. The apparatus of claim 1 , wherein the instructions, when executed, cause the one or more processors to:
receive model telemetry data periodically, the model telemetry data to include one or more of incoming field data, a signal data output by model, or a user response to a prompt; and
determine whether a data distribution of the impact of the feature candidates on the target malware prediction deviates from the model telemetry data; and
retrain the impact of the feature candidates on the target malware prediction in response to the data distribution of the impact of the feature candidates on the target malware prediction deviating from the model telemetry data.
7. An apparatus to detect and halt malware, the apparatus comprising:
memory; and
one or more processors to execute instructions that cause the one or more processors to:
analyze an impact of feature candidates on a target malware prediction, the feature candidates to be associated with a predetermined portion of an original or modified file associated with a file system event;
determine, using a design of experiments, singular or combinatorial features to be associated with the file system event based on the feature candidates;
determine a malware prediction based on the features associated with the file system event, the malware prediction associated with a predetermined operation; and
evaluate test data to determine whether the features associated with the file system event overfit training data, the features associated with the file system event to overfit the training data in response to an accuracy of the malware prediction being higher for the training data compared to the test data.
8. The apparatus of claim 7 , wherein the instructions, when executed, cause the one or more processors to assign weights to the features associated with the file system event based on the impact of the feature candidates on the target malware prediction.
9. A method comprising:
analyzing an impact of feature candidates on a target malware prediction, the feature candidates to be associated with a predetermined portion of an original or modified file associated with a file system event;
determining singular or combinatorial features to be associated with the file system event based on the feature candidates;
determining a malware prediction based on the features associated with the file system event, the malware prediction associated with a predetermined operation;
receiving model telemetry data periodically, the model telemetry data including one or more of incoming field data, a signal data output by model, or a user response to a prompt;
determining whether a data distribution of the impact of the feature candidates on the target malware prediction deviates from the model telemetry data; and
retraining the impact of the feature candidates on the target malware prediction in response to the data distribution of the impact of the feature candidates on the target malware prediction deviating from the model telemetry data.
10. The method of claim 9 , further including assigning weights to the features associated with the file system event based on the impact of the feature candidates on the target malware prediction.
11. The method of claim 9 , wherein the predetermined portion of the original or modified file is associated with at least one of a first 4k bits of the original file, a first 4k bits of the modified file, a last 4k bits associated with the original file, a last 4k bits associated with the modified file.
12. The method of claim 11 , wherein the feature candidates include an entropy of the predetermined portion of the original or modified file, a Monte Carlo pi estimate obtained based on the predetermined portion of the original or modified file, a Monte Carlo pi estimation error based on the predetermined portion of the original or modified file, a serial correlation coefficient associated with the predetermined portion of the original or modified file, a chi square associated with the original or modified file, or an arithmetic mean associated with the original or modified file.
13. The method of claim 9 , further including evaluating test data to determine whether the features associated with the file system event overfit training data, the features associated with the file system event to overfit the training data in response to an accuracy of the malware prediction being higher for the training data compared to the test data.
14. The method of claim 9 , further including:
replacing missing or invalid data associated with the file system event;
determining Cartesian products between the feature candidates; and
performing non-linear transformations associated with the feature candidates.
15. A non-transitory computer readable medium including computer executable instructions that, when executed, cause one or more processing units to at least:
analyze an impact of feature candidates on a target malware prediction, the feature candidates to be associated with a predetermined portion of an original or modified file associated with a file system event;
determine singular or combinatorial features to be associated with the file system event based on the feature candidates;
determine a malware prediction based on the features associated with the file system event, the malware prediction associated with a predetermined operation;
replace missing or invalid data associated with the file system event;
determine Cartesian products between the feature candidates; and
perform non-linear transformations associated with the feature candidates.
16. The non-transitory computer readable medium of claim 15 , wherein the instructions, when executed, cause the one or more processing units to assign weights to the features associated with the file system event based on the impact of the feature candidates on the target malware prediction.
17. The non-transitory computer readable medium of claim 15 , wherein the predetermined portion of the original or modified file is associated with at least one of a first 4k bits of the original file, a first 4k bits of the modified file, a last 4k bits associated with the original file, a last 4k bits associated with the modified file.
18. The non-transitory computer readable medium of claim 17 , wherein the feature candidates include an entropy of the predetermined portion of the original or modified file, a Monte Carlo pi estimate obtained based on the predetermined portion of the original or modified file, a Monte Carlo pi estimation error based on the predetermined portion of the original or modified file, a serial correlation coefficient associated with the predetermined portion of the original or modified file, a chi square associated with the original or modified file, or an arithmetic mean associated with the original or modified file.
19. The non-transitory computer readable medium of claim 15 , wherein the instructions, when executed, cause the one or more processing units to evaluate test data to determine whether the features associated with the file system event overfit training data, the features associated with the file system event to over-fit the training data in response to an accuracy of the malware prediction being higher for the training data compared to the test data.
20. The non-transitory computer readable medium of claim 15 , wherein the instructions, when executed, cause the one or more processing units to:
receive model telemetry data periodically, the model telemetry data including one or more of incoming field data, a signal data output by model, or a user response to a prompt;
determine whether a data distribution of the impact of the feature candidates on the target malware prediction deviates from the model telemetry data; and
retrain the impact of the feature candidates on the target malware prediction in response to the data distribution of the impact of the feature candidates on the target malware prediction deviating from the model telemetry data.