Detection of malicious executable files using hierarchical models

1. A method comprising:

disassembling, by a device, an executable file into assembly instructions;

mapping, by the device, each of the assembly instructions to a fixed length instruction vector using one-hot encoding and an instruction vocabulary;

forming, by the device, vector representations of blocks of a control flow graph for corresponding functions of the executable file by embedding and aggregating bags of the instruction vectors;

generating, by the device and based on the formed vector representations of the blocks of the control flow graph, a call graph model of the functions in the executable file;

forming, by the device, a vector representation of the executable file based in part on the call graph model; and

determining, by the device and based on the vector representation of the executable file, whether the executable file is malware.

2. The method as in claim 1 , further comprising:

initiating a corrective measure when the device determines that the executable file is malware.

3. The method as in claim 1 , wherein the device uses a machine learning classifier to determine that the executable file is malware.

4. The method as in claim 1 , wherein forming vector representations of blocks of the control flow graph for the executable file by embedding and aggregating bags of the instruction vectors comprises:

concatenating the instruction vectors into bags; and

processing the concatenated vectors using a neural network with aggregation.

5. The method as in claim 1 , wherein forming vector representations of blocks of the control flow graph for the executable file by embedding and aggregating bags of the instruction vectors comprises:

computing a histogram of n-grams using the instruction vectors, to form the vector representations of blocks of the control flow graph.

6. The method as in claim 1 , wherein forming the vector representation of the executable file based in part on the call graph model of the executable file comprises:

combining the call graph model with data regarding a portable executable (PE) header of the executable file.

7. The method as in claim 1 , wherein forming vector representations of blocks of the control flow graph for the executable file by embedding and aggregating bags of the instruction vectors comprises:

combining the instruction vectors with any associated string references or call names from the executable file.

8. The method as in claim 1 , wherein generating, based on the formed vector representations of the blocks of the control flow graph, the call graph model of the executable file comprises:

forming the control flow graph using the vector representations of the blocks of the control flow graph by constructing a block wrapper for the particular block using the vector representations of its incoming and outgoing blocks.

9. An apparatus, comprising:

one or more network interfaces to communicate with a network;

a processor coupled to the network interfaces and configured to execute one or more processes; and

a memory configured to store a process executable by the processor, the process when executed configured to:

disassemble an executable file into assembly instructions;

map each of the assembly instructions to a fixed length instruction vector using one-hot encoding and an instruction vocabulary;

form vector representations of blocks of a control flow graph for corresponding functions of the executable file by embedding and aggregating bags of the instruction vectors;

generate, based on the formed vector representations of the blocks of the control flow graph, a call graph model of the functions in the executable file;

form, a vector representation of the executable file based in part on the call graph model of the executable file; and

determine, based on the vector representation of the executable file, whether the executable file is malware.

10. The apparatus as in claim 9 , wherein the process when executed is further configured to:

initiate a corrective measure when the executable file is determined to be malware.

11. The apparatus as in claim 9 , wherein the apparatus uses a machine learning classifier to determine that the executable file is malware.

12. The apparatus as in claim 9 , wherein the apparatus forms vector representations of blocks of the control flow graph for corresponding functions of the executable file by embedding and aggregating bags of the instruction vectors by:

concatenating the instruction vectors into bags; and

processing the concatenated vectors using a neural network with aggregation.

13. The apparatus as in claim 9 , wherein the apparatus forms vector representations of blocks of the control flow graph for the executable file by embedding and aggregating bags of the instruction vectors by:

computing a histogram of n-grams using the instruction vectors, to form the vector representations of blocks of the control flow graph.

14. The apparatus as in claim 9 , wherein the apparatus forms the vector representation of the executable file based in part on the call graph model of the executable file by:

combining the call graph model with data regarding a portable executable (PE) header of the executable file.

15. The apparatus as in claim 9 , wherein the apparatus forms vector representations of blocks of the control flow graph for the executable file by embedding and aggregating bags of the instruction vectors by:

combining the instruction vectors with any associated string references or call names from the executable file.

16. The apparatus as in claim 9 , wherein the apparatus generates, based on the formed vector representations of the blocks of the control flow graph, the call graph model of the executable file by:

forming the control flow graph using the vector representations of the blocks of the control flow graph by constructing a block wrapper for the particular block using the vector representations of its incoming and outgoing blocks.

17. A tangible, non-transitory, computer-readable medium storing program instructions that cause a device to execute a process comprising:

disassembling, by a device, an executable file into assembly instructions;

mapping, by the device, each of the assembly instructions to a fixed length instruction vector using one-hot encoding and an instruction vocabulary;

forming, by the device, vector representations of blocks of a control flow graph for corresponding functions the executable file by embedding and aggregating bags of the instruction vectors;

generating, by the device and based on the formed vector representations of the blocks of the control flow graph, a call graph model of the functions in the executable file;

forming, by the device, a vector representation of the executable file based in part on the call graph model; and

determining, by the device and based on the vector representation of the executable file, whether the executable file is malware.

18. The computer-readable medium as in claim 17 , further comprising:

initiating a corrective measure when the device determines that the executable file is malware.

19. The computer-readable medium as in claim 17 , wherein forming vector representations of blocks of the control flow graph for corresponding functions of the executable file by embedding and aggregating bags of the instruction vectors comprises:

concatenating the instruction vectors into bags; and

processing the concatenated vectors using a neural network with aggregation.

20. The computer-readable medium as in claim 17 , wherein forming vector representations of blocks of the control flow graph for the executable file by embedding and aggregating bags of the instruction vectors comprises:

computing a histogram of n-grams using the instruction vectors, to form the vector representations of blocks of the control flow graph.