METHODS AND SYSTEMS FOR PER-RESOURCE ANOMALY DETECTION
Disclosed herein are system, method, and computer program product embodiments for detecting ransomware and creating ransomware incidents by way of analyzing for ransomware signals in a backup data stream on a file-by-file basis. The ransomware detection system comprises a ransomware detection engine that includes a tracking component. The tracking component may track the byte distribution and extension of a file from a backup data stream. Further, the tracking component may perform a ransomware analysis on the file and identify, using a machine learning model, that the file is encrypted by ransomware based on an anomaly score and a confidence threshold. Subsequently, the tracking component may create a ransomware incident based on the identification that the file is encrypted by ransomware. Disclosed herein are additional embodiments directed towards training and updating a machine learning model within the ransomware detection engine.
1 . A computer-implemented method for detecting a ransomware incident in a backup data stream, the computer-implemented method comprising:
tracking a byte distribution of a file from the backup data stream;
calculating an entropy of the byte distribution of the file, wherein the entropy is reflected by an anomaly score for the file;
mapping the file to a confidence threshold of a plurality of confidence thresholds based on a set of file characteristics of the file;
identifying, by a machine learning model, that the file is encrypted by ransomware based on the anomaly score of the file exceeding the confidence threshold; and
creating the ransomware incident based on the identifying that the file is encrypted by the ransomware.
2 . The computer-implemented method of claim 1 , further comprising:
aggregating metadata of the file, wherein the metadata comprises file size, file extension, file type, or the anomaly score of the file;
classifying the aggregated metadata of the file into a bucket of a plurality of buckets based on one or more file characteristics selected from the set of file characteristics, wherein the bucket comprises aggregated metadata of another file; and
training the machine learning model using at least the aggregated metadata in the bucket.
3 . The computer-implemented method of claim 1 , wherein the identifying comprises:
evaluating the file for a particular ransomware attack of a set of ransomware attacks, wherein the set of ransomware attacks comprises in-place encryption ransomware or delete-and-replace ransomware.
4 . The computer-implemented method of claim 1 , wherein the machine learning model comprises a random cut forest model.
5 . The computer-implemented method of claim 1 , wherein the calculating comprises:
applying a chi square entropy scoring equation on the byte distribution.
6 . The computer-implemented method of claim 1 , wherein the mapping comprises:
correlating the file to a resource type using the set of file characteristics, wherein the set of file characteristics comprises file type, file size, file age, file extension, or file usage; and
extracting the confidence threshold corresponding to the resource type.
7 . The computer-implemented method of claim 6 , wherein the extracting comprises:
calculating an average anomaly score for one or more files correlated to the resource type over a number of previous days; and
defining the confidence threshold to be within a number of standard deviations from the average anomaly score.
8 . A system for detecting a ransomware incident in a backup data stream, the system comprising:
one or more memories;
at least one processor each coupled to at least one of the memories and configured to perform operations comprising:
tracking a byte distribution of a file from the backup data stream;
calculating an entropy of the byte distribution of the file, wherein the entropy is reflected by an anomaly score for the file;
mapping the file to a confidence threshold of a plurality of confidence thresholds based on a set of file characteristics of the file;
identifying, by a machine learning model, that the file is encrypted by ransomware based on the anomaly score of the file exceeding the confidence threshold; and
creating the ransomware incident based on the identifying that the file is encrypted by the ransomware.
9 . The system of claim 8 , wherein the operations further comprise:
aggregating metadata of the file, wherein the metadata comprises file size, file extension, file type, or the anomaly score of the file;
classifying the aggregated metadata of the file into a bucket of a plurality of buckets based on one or more file characteristics selected from the set of file characteristics, wherein the bucket comprises aggregated metadata of another file; and
training the machine learning model using at least the aggregated metadata in the bucket.
10 . The system of claim 8 , wherein the identifying comprises:
evaluating the file for a particular ransomware attack of a set of ransomware attacks, wherein the set of ransomware attacks comprises in-place encryption ransomware or delete-and-replace ransomware.
11 . The system of claim 8 , wherein the machine learning model comprises a random cut forest model.
12 . The system of claim 8 , wherein the calculating comprises:
applying a chi square entropy scoring equation on the byte distribution.
13 . The system of claim 8 , wherein the mapping comprises:
correlating the file to a resource type using the set of file characteristics, wherein the set of file characteristics comprises file type, file size, file age, file extension, or file usage; and
extracting the confidence threshold corresponding to the resource type.
14 . The system of claim 13 , wherein the extracting comprises:
calculating an average anomaly score for one or more files correlated to the resource type over a number of previous days; and
defining the confidence threshold to be within a number of standard deviations from the average anomaly score.
15 . A non-transitory computer-readable device having instructions stored thereon that, when executed by at least one computing device, causes the at least one computing device to perform operations for detecting a ransomware incident in a backup data stream, the operations comprising:
tracking a byte distribution of a file from the backup data stream;
calculating an entropy of the byte distribution of the file, wherein the entropy is reflected by an anomaly score for the file;
mapping the file to a confidence threshold of a plurality of confidence thresholds based on a set of file characteristics of the file;
identifying, by a machine learning model, that the file is encrypted by ransomware based on the anomaly score of the file exceeding the confidence threshold; and
creating the ransomware incident based on the identifying that the file is encrypted by the ransomware.
16 . The non-transitory computer readable device of claim 15 , the operations further comprising:
aggregating metadata of the file, wherein the metadata comprises file size, file type, file extension, or the anomaly score of the file;
classifying the aggregated metadata of the file into a bucket of a plurality of buckets based on one or more file characteristics selected from the set of file characteristics, wherein the bucket comprises aggregated metadata of another file; and
training the machine learning model using at least the aggregated metadata in the bucket.
17 . The non-transitory computer readable device of claim 15 , wherein the identifying comprises:
evaluating the file for a particular ransomware attack of a set of ransomware attacks, wherein the set of ransomware attacks comprises in-place encryption ransomware or delete-and-replace ransomware.
18 . The non-transitory computer readable device of claim 15 , wherein the machine learning model comprises a random cut forest model.
19 . The non-transitory computer readable device of claim 15 , wherein the calculating comprises:
applying a chi square entropy scoring equation on the byte distribution.
20 . The non-transitory computer readable device of claim 15 , wherein the mapping comprises:
correlating the file to a resource type using the set of file characteristics, wherein the set of file characteristics comprises file type, file size, file age, file extension, or file usage; and
extracting the confidence threshold corresponding to the resource type.