Malware fingerprinting on encrypted transport layer security (TLS) traffic

1. A computer-implemented method for malware fingerprinting on encrypted transport layer security (TLS) traffic, at least a portion of the method being performed by a computing device comprising one or more processors, the method comprising:

obtaining, by the computing device, input data of a TLS communication between a client device and a server device;

extracting, by the computing device, features associated with the TLS communication;

determining, by the computing device, an association between the TLS communication and a known malware family by applying a clustering algorithm to the extracted features;

analyzing, by the computing device, at least one cluster determined by applying the clustering algorithm to determine a purity level;

assigning, by the computing device, a malware family fingerprint to the TLS communication based on the purity level;

applying, by the computing device, the malware family fingerprint to a backend of a network to determine a probability of an attack; and

initiating, by the computing device, a security action based on the probability of the attack.

2. The method of claim 1 , wherein the initiating of the security action further comprises:

initiating one or more of isolating the client device, isolating the server device, transmitting a message to an administrator, or applying a malware family-specific security action to a TLS connection identified as malicious, or a combination thereof.

3. The method of claim 1 , further comprising:

normalizing the input data of the TLS communication using a term frequency-inverse document frequency (TF-IDF) model.

4. The method of claim 1 , further comprising:

associating the TLS communication with a malware family based on identifying a similarity of features between the TLS communication and communications associated with the malware family.

5. The method of claim 1 , wherein the obtaining of input data further comprises:

executing known instances of malware in a sandbox for a pre-determined time period;

collecting data regarding network traffic of the known instances of malware; and

collecting data regarding a TLS handshake of the known instances of malware.

6. The method of claim 1 , wherein the extracting further comprises:

extracting from an unencrypted portion of the TLS communication one or more of a server validation status, a server name, a compression method of a TLS version, ciphers supported by the client device, ciphers chosen by the server device, TLS extensions supported by the server device, TLS extensions supported by the client device, a server certificate validity, a client device certificate validity, whether a certificate is self-signed, or a number of client certificates in a chain number of server certificates, or a combination thereof.

7. The method of claim 1 , wherein the extracting further comprises:

extracting from an encrypted portion of the TLS communication one or more of a number of encrypted bytes sent and received, a number of encrypted packets sent and received, a duration of encrypted traffic sent and received, a ratio of data sent and received, an average packet size, a maximum contiguous packet sequence length, a packet size distribution, or interarrival patterns, or a combination thereof.

8. The method of claim 1 , wherein the applying of the clustering algorithm further comprises:

applying a distance function on the TLS communication.

9. The method of claim 1 , wherein the determining of the purity level further comprises:

determining that a similarity between the TLS communication and a communication by a known malware family satisfies a pre-determined threshold.

10. The method of claim 1 , further comprising:

before applying the malware family fingerprint to the backend, applying the malware family fingerprint to a known-benign TLS network to identify benign TLS connections; and

applying the malware family fingerprint to connections not identified as benign TLS connections.

11. One or more non-transitory computer-readable media comprising one or more computer readable instructions that, when executed by one or more processors of a computing device cause the computing device to perform a method for malware fingerprinting on encrypted transport layer security (TLS) traffic, the method comprising:

obtaining, by the computing device, input data of a TLS communication between a client device and a server device;

extracting, by the computing device, features associated with the TLS communication;

determining, by the computing device, an association between the TLS communication and a known malware family by applying a clustering algorithm to the extracted features;

analyzing, by the computing device, at least one cluster determined by applying the clustering algorithm to determine a purity level;

assigning, by the computing device, a malware family fingerprint to the TLS communication based on the purity level;

applying, by the computing device, the malware family fingerprint to a backend of a network to determine a probability of an attack; and

initiating, by the computing device, a security action based on the probability of the attack.

12. The one or more non-transitory computer-readable media of claim 11 , further comprising:

normalizing the input data of the TLS communication using a term frequency-inverse document frequency (TF-IDF) model.

13. The one or more non-transitory computer-readable media of claim 11 , further comprising:

associating the TLS communication with a malware family based on identifying a similarity of features between the TLS communication and communications associated with the malware family.

14. The one or more non-transitory computer-readable media of claim 11 , wherein the obtaining of input data further comprises:

executing known instances of malware in a sandbox for a pre-determined time period;

collecting data regarding network traffic of the known instances of malware; and

collecting data regarding the TLS communication of the known instances of malware.

15. The one or more non-transitory computer-readable media of claim 11 , wherein the extracting further comprises:

extracting from an unencrypted portion of the TLS communication one or more of a server validation status, a server name, a compression method of a TLS version, ciphers supported by the client device, ciphers chosen by the server device, TLS extensions supported by the server device, TLS extensions supported by the client device, a server certificate validity, a client device certificate validity, whether a certificate is self-signed, or a number of client certificates in a chain number of server certificates, or a combination thereof.

16. The one or more non-transitory computer-readable media of claim 11 , wherein the extracting further comprises:

extracting from an encrypted portion of the TLS communication one or more of a number of encrypted bytes sent and received, a number of encrypted packets sent and received, a duration of encrypted traffic sent and received, a ratio of data sent and received, an average packet size, a maximum contiguous packet sequence length, a packet size distribution, or interarrival patterns, or a combination thereof.

17. The one or more non-transitory computer-readable media of claim 11 , wherein the applying the clustering algorithm further comprises:

applying a distance function on the TLS communication.

18. The one or more non-transitory computer-readable media of claim 11 , wherein the determining of the purity level further comprises:

determining that a similarity between the TLS communication and a communication by a known malware family satisfies a pre-determined threshold.

19. The one or more non-transitory computer-readable media of claim 11 , further comprising:

before applying the malware family fingerprint to the backend, applying the malware family fingerprint to a known-benign TLS network to identify benign TLS connections; and

applying the malware family fingerprint to connections not identified as benign TLS connections.

20. A system for determining for malware fingerprinting on encrypted transport layer security (TLS) traffic, the system comprising:

a processor;

a memory in electronic communication with the processor; and

instructions stored in the memory, the instructions being executable by the processor to:

obtain input data of a TLS communication between a client device and a server device;

extract, by the computing device, features associated with the TLS communication;

determine an association between the TLS communication and a known malware family by applying a clustering algorithm to the extracted features;

analyze at least one cluster determined by applying the clustering algorithm to determine a purity level;

assign a malware family fingerprint to the TLS communication based on the purity level;

apply the malware family fingerprint to a backend of a network to determine a probability of an attack; and

initiate a security action based on the probability of the attack.