Machine learning-based encrypted file classification for identifying encrypted data movement
The disclosed technology facilitates User and Entity Behavior Analytics (UEBA) by classifying a file being transferred as encrypted or not. The technology involves monitoring movement of a files by a user over a wide area network, detecting file encryption for the files using a trained classifier, wherein the detecting includes processing by the classifier some or all of the following features extracted from each of the files: a chi-square randomness test; an arithmetic mean test; a serial correlation coefficient test; a Monte Carlo-Pi test; and a Shannon entropy test, counting a number of the encrypted files moved by the user in a predetermined period, comparing a predetermined maximum number of encrypted files allowed in the predetermined period to the count of the encrypted files moved by the user and detecting that the user has moved more encrypted files than the predetermined maximum number, and generating an alert.
1 . A computer-implemented method, comprising:
intercepting, by a network security system server interposed on a network between a cloud-based application and a user endpoint associated with a user, movement of a plurality of files over the network to the cloud-based application;
classifying, with a classifier of the network security system server, each file of the plurality of files as one of encrypted or unencrypted, wherein:
the classifier is trained to classify each file based on analyzing sampled bytes of the respective file and a file type of the respective file; and
the classifying comprises:
determining the file type of the respective file;
calculating metrics for the respective file based on analyzing the sampled bytes; and
providing the metrics and the file type to the classifier trained to classify the respective file as encrypted or unencrypted based on the metrics and the file type;
counting, by the network security system server, a number of the plurality of files classified as encrypted and moved by the user during a predetermined time period;
detecting, by the network security system server, based on the counting, that the user has moved more encrypted files than a predetermined maximum number of encrypted files the user is allowed to move during the predetermined time period; and
generating, by the network security system server in response to the detecting, an alert that the user has moved more encrypted files than the user is allowed.
2 . The computer-implemented method of claim 1 , wherein the calculating the metrics comprises calculating two or metrics selected from a chi-square metric, an arithmetic mean metric, a serial correlation coefficient metric, a Monte Carlo-Pi metric, and an entropy metric.
3 . The computer-implemented method of claim 2 , wherein:
the chi-square metric is based on a chi-square randomness test that measures a degree to which a distribution of the sampled bytes varies from an expected distribution of bytes from the respective file;
the arithmetic mean metric is based on an arithmetic mean test that compares an arithmetic mean of the sampled bytes to an expected mean of the bytes from the respective file;
the serial correlation coefficient metric is based on a serial correlation coefficient test that calculates a serial correlation coefficient between pairs of successive sampled bytes from the respective file;
the Monte Carlo-Pi metric is based on a Monte Carlo-Pi test that maps concatenated bytes as coordinates of a square and calculates a degree to which a proportion of the mapped concatenated bytes that fall within a circle circumscribed by the square varies from an expected proportion that corresponds to mapping from the respective file; and
the entropy metric is based on a Shannon entropy test of randomness of the respective file.
4 . The computer-implemented method of claim 1 , further comprising:
analyzing file movement data for the user over a number of days; and
establishing the predetermined maximum number of encrypted files allowed to be moved based on the analyzing.
5 . The computer-implemented method of claim 4 , wherein the number of days is at least 15 days.
6 . The computer-implemented method of claim 1 , further comprising:
analyzing file movement data for a plurality of users within an organization over a number of user-days; and
establishing the predetermined maximum number of encrypted files allowed to be moved based on the analyzing.
7 . The computer-implemented method of claim 6 , wherein the number of user-days is at least one thousand (1000).
8 . The computer-implemented method of claim 7 , wherein the number of user-days includes both workdays and non-workdays.
9 . The computer-implemented method of claim 1 , further comprising:
intercepting first movement of a first subset of the plurality of files over an application programming interface (API) connection to the cloud-based application; and
intercepting second movement of a second subset of the plurality of files via inline traffic associated with the user endpoint.
10 . The computer-implemented method of claim 1 , further comprising:
sampling the respective file to obtain the sampled bytes, wherein a size of the sampled bytes comprises between 10 KB and 250 KB of the respective file.
11 . A network security system interposed on a network between a cloud-based application and a user endpoint associated with a user, the network security system comprising:
one or more processors; and
a memory having stored thereon instructions that, upon execution by the one or more processors, cause the one or more processors to:
intercept movement of a plurality of files over the network to the cloud-based application;
classify, with a classifier, each file of the plurality of files as one of encrypted or unencrypted, wherein;
the classifier is trained to classify each file based on analyzing sampled bytes of the respective file and a file type of the respective file; and
the instructions to classify comprise further instructions that, upon execution by the one or more processors, cause the one or more processors to:
determine the file type of the respective file;
calculate metrics for the respective file based on analyzing the sampled bytes; and
provide the metrics and the file type to the classifier trained to classify the respective file as encrypted or unencrypted based on the metrics and the file type;
count a number of the plurality of files classified as encrypted and moved by the user during a predetermined time period;
detect, based on the counting, that the user has moved more encrypted files than a predetermined maximum number of encrypted files the user is allowed to move during the predetermined time period; and
generate, in response to the detecting, an alert that the user has moved more encrypted files than the user is allowed.
12 . The network security system of claim 11 , wherein the instructions to calculate comprise further instructions that, upon execution by the one or more processors, cause the one or more processors to calculate two or more metrics selected from a chi-square metric, an arithmetic mean metric, a serial correlation coefficient metric, a Monte Carlo-Pi metric, and an entropy metric.
13 . The network security system of claim 12 , wherein:
the chi-square metric is based on a chi-square randomness test that measures a degree to which a distribution of the sampled bytes varies from an expected distribution of bytes from the respective file;
the arithmetic mean metric is based on an arithmetic mean test that compares an arithmetic mean of the sampled bytes to an expected mean of the bytes from the respective file;
the serial correlation coefficient metric is based on a serial correlation coefficient test that calculates a serial correlation coefficient between pairs of successive sampled bytes from the respective file;
the Monte Carlo-Pi metric is based on a Monte Carlo-Pi test that maps concatenated bytes as coordinates of a square and calculates a degree to which a proportion of the mapped concatenated bytes that fall within a circle circumscribed by the square varies from an expected proportion that corresponds to mapping from the respective file; and
the entropy metric is based on a Shannon entropy test of randomness of the respective file.
14 . The network security system of claim 11 , wherein the instructions comprise further instructions that, upon execution by the one or more processors, cause the one or more processors to:
analyze file movement data for the user over a number of days; and
establish the predetermined maximum number of encrypted files allowed to be moved based on the analyzing.
15 . The network security system of claim 14 , wherein the number of days is at least 15 days.
16 . The network security system of claim 11 , wherein the instructions comprise further instructions that, upon execution by the one or more processors, cause the one or more processors to:
analyze file movement data for a plurality of users within an organization over a number of user-days; and
establish the predetermined maximum number of encrypted files allowed to be moved based on the analyzing.
17 . The network security system of claim 16 , wherein the number of user-days is at least one thousand (1000).
18 . The network security system of claim 17 , wherein the number of user-days includes both workdays and non-workdays.
19 . The network security system of claim 11 , wherein the instructions comprise further instructions that, upon execution by the one or more processors, cause the one or more processors to:
intercept first movement of a first subset of the plurality of files over an application programming interface (API) connection to the cloud-based application; and
intercept second movement of a second subset of the plurality of files via inline traffic associated with the user endpoint.
20 . The network security system of claim 11 , wherein the instructions comprise further instructions that, upon execution by the one or more processors, cause the one or more processors to:
sample the respective file to obtain the sampled bytes, wherein a size of the sampled bytes comprises between 10 KB and 250 KB of the respective file.