Systems and methods for structural similarity based hashing
Systems and methods for structural similarity based hash for sample identification and detection include, monitoring traffic associated with a cloud-based system; identifying a unique file within the traffic and computing a Structural Similarity Hash (SSHash) for the file, wherein the SSHash is based on auxiliary information and a complexity of the file; identifying one or more similar files based on the SSHash; and defining the file as belonging to one or more groups based on the one or more similar files.
1 . A non-transitory computer-readable medium having instructions stored thereon for programming one or more processors to perform steps of:
monitoring traffic associated with a cloud-based system;
identifying a unique file within the traffic and computing a Structural Similarity Hash (SSHash) for the file, wherein computing the SSHash comprises splitting the filed into a plurality of chunks, computing a complexity of each of the plurality of chunks, rounding the computed complexities to reduce noise, collecting auxiliary information including at least a file type, chunk size, and number of chunks, and generating the SSHash based on both (i) the distribution density of the computed complexities across the plurality of chunks and (ii) the auxiliary information;
identifying one or more similar files based on the SSHash by referencing a database of SSHashes, the database being populated through continuous learning including enrichment from sandbox verdicts of unknown files and curated sets of known clean and known malicious files; and
defining the file as belonging to one of multiple groups comprising at least malware, clean, or unknown based on the one or more similar files.
2 . The non-transitory computer-readable medium of claim 1 , wherein computing the SSHash further comprises:
splitting the file into one or more equal length chunks;
computing a complexity of each of the one or more equal length chunks, including determining a Kolmogorov-style complexity measure of each chunk;
rounding the computed complexity values to mitigate noise and calculating a distribution density of the complexity values across the chunks;
collecting auxiliary information from the file; and
generating the SSHash based on the auxiliary information and the distribution density of the complexity of the file.
3 . The non-transitory computer-readable medium of claim 1 , wherein responsive to determining that no similar files exist, sending the unique file to a sandbox for analysis.
4 . The non-transitory computer-readable medium of claim 1 , wherein identifying one or more similar files includes referencing a SSHash database of files having known groupings.
5 . The non-transitory computer-readable medium of claim 4 , wherein the database is managed and persisted in a Central Database Management System (CDMS) of the cloud-based system.
6 . The non-transitory computer-readable medium of claim 5 , wherein the monitoring and computing are performed via one or more nodes of the cloud-based system communicatively coupled to the CDMS, and wherein the one or more nodes of the cloud-based system are adapted to send query requests to the CDMS for identifying one or more similar files.
7 . The non-transitory computer-readable medium of claim 1 , wherein the steps further comprise:
performing an action based on the defining, wherein the action includes one of blocking the traffic and allowing the traffic.
8 . The non-transitory computer-readable medium of claim 1 , wherein the steps comprise:
building an SSHash database prior to the monitoring.
9 . The non-transitory computer-readable medium of claim 8 , wherein building the SSHash database includes sending a plurality of sample files to a sandbox for grouping the plurality of sample files.
10 . The non-transitory computer-readable medium of claim 1 , wherein the unique file is any of a P file, a Portable Executable (PE) file, a shortcut (LNK) file, a Portable Document Format (PDF) file, an Executable Linkable Format (ELF) file, Android Package Kit (APK) file, Hypertext Markup Language (HTML) file.
11 . A method comprising steps of:
monitoring traffic associated with a cloud-based system;
identifying a unique file within the traffic and computing a Structural Similarity Hash (SSHash) for the file, wherein computing the SSHash comprises splitty the file into a plurality of chunks, computing a complexity of each of the plurality of chunks, rounding the computed complexities to reduce noise, collecting auxiliary information including at least a file type, chunk size, and number of chunks, and generating the SSHash based on both (i) the distribution density of the computed complexities across the plurality of chunks and (ii) the auxiliary information;
identifying one or more similar files based on the SSHash by referencing a database of SSHashes, the database being populated through continuous learning including enrichment from sandbox verdicts of unknown files and curated sets of known clean and known malicious files; and
defining the file as belonging to one of multiple groups comprising at least malware, clean, or unknown based on the one or more similar files.
12 . The method of claim 11 , wherein computing the SSHash further comprises:
splitting the file into one or more equal length chunks;
computing a complexity of each of the one or more equal length chunks, including determining a Kolmogorov-style complexity measure of each chunk;
rounding the computed complexity values to mitigate noise and calculating a distribution density of the complexity values across the chunks;
collecting auxiliary information from the file; and
generating the SSHash based on the auxiliary information and the distribution density of the complexity of the file.
13 . The method of claim 11 , wherein responsive to determining that no similar files exist, sending the unique file to a sandbox for analysis.
14 . The method of claim 11 , wherein identifying one or more similar files includes referencing a SSHash database of files having known groupings.
15 . The method of claim 14 , wherein the database is managed and persisted in a Central Database Management System (CDMS) of the cloud-based system.
16 . The method of claim 15 , wherein the monitoring and computing are performed via one or more nodes of the cloud-based system communicatively coupled to the CDMS, and wherein the one or more nodes of the cloud-based system are adapted to send query requests to the CDMS for identifying one or more similar files.
17 . The method of claim 11 , wherein the steps further comprise:
performing an action based on the defining, wherein the action includes one of blocking the traffic and allowing the traffic.
18 . The method of claim 11 , wherein the steps comprise:
building an SSHash database prior to the monitoring.
19 . The method of claim 18 , wherein building the SSHash database includes sending a plurality of sample files to a sandbox for grouping the plurality of sample files.
20 . The method of claim 11 , wherein the unique file is any of a P file, a Portable Executable (PE) file, a shortcut (LNK) file, a Portable Document Format (PDF) file, an Executable Linkable Format (ELF) file, Android Package Kit (APK) file, Hypertext Markup Language (HTML) file.