Hardware-enabled prevention of code reuse attacks
Described systems and methods allow protecting a host computer system from malware, such as return-oriented programming (ROP) and jump-oriented programming (JOP) exploits. In some embodiments, a processor of the host system is endowed with two counters configured to store a count of branch instructions and a count of inter-branch instructions, respectively, occurring within a stream of instructions fetched by the processor for execution. Exemplary counted branch instructions include indirect JMP, indirect CALL, and RET on x86 platforms, while inter-branch instructions consist of instructions executed between two consecutive counted branch instructions. The processor may be further configured to generate a processor event, such as an exception, when a value stored in a counter exceeds a predetermined threshold. Such events may be used as triggers for launching a malware analysis to determine whether the host system is subject to a code reuse attack.
1. A host system comprising a hardware processor, the hardware processor including:
a branch counter register configured to store a count of branch instructions occurring within a sequence of instructions executed by the hardware processor;
an inter-branch instruction counter register configured to store a count of instructions occurring between two consecutive branch instructions within the sequence of instructions; and
a counter control unit connected to the branch counter register, to the inter-branch instruction counter register, and to an instruction decoder module of the hardware processor,
wherein the counter control unit comprises hardware logic configured to:
perform branch monitoring, and
trigger a switch event within the hardware processor according to a value stored in the branch counter register and according to a value stored in the inter-branch instruction counter register, wherein the switch event causes the hardware processor to switch from executing the sequence of instructions to executing an event handler routine,
wherein branch monitoring comprises:
determining whether a selected instruction of the sequence of instructions is a branch instruction;
in response to determining whether the selected instruction is a branch instruction, when the selected instruction is a branch instruction, incrementing the value stored in the branch counter register; and
in response to determining whether the selected instruction is a branch instruction, when the selected instruction is not a branch instruction, incrementing the value stored in the inter-branch instruction counter register;
and wherein the hardware processor is further configured, in response to the switch event, to:
identify a target execution thread according to the sequence of instructions; and
in response to identifying the target execution thread, determine whether the target execution thread is performing a code-reuse attack.
2. The host system of claim 1 , wherein the counter control unit is further configured to:
compare the value stored in the branch counter register to a predetermined threshold; and
in response, determine whether to trigger the switch event according to a result of the comparison.
3. The host system of claim 1 , wherein the counter control unit is further configured to:
compare the value stored in the inter-branch instruction counter register to a predetermined threshold; and
in response, when the value stored in the inter-branch instruction counter register exceeds a predetermined threshold, reset the value stored in the branch counter register to zero.
4. The host system of claim 1 , wherein the switch event comprises an exception, and wherein the event handler routine comprises an exception handler registered in an interrupt descriptor table (IDT) of the processor.
5. The host system of claim 1 , wherein the host system exposes a virtual machine, wherein the sequence of instructions consists of code executing within the virtual machine, and wherein the event handler routine executes outside the virtual machine.
6. The host system of claim 1 , wherein branch instructions include at least one instruction selected from a group consisting of a jump instruction, a call instruction, and a return instruction.
7. The host system of claim 1 , wherein the hardware logic of the counter control unit is further configured, when the selected instruction is a branch instruction, to reset the value stored in the inter-branch instruction counter register to zero.
8. The host system of claim 1 , wherein the hardware processor further includes a register having a branch identifier field, and wherein the counter control unit is configured to determine whether the selected instruction is a branch instruction according to a value stored in a branch identifier field.
9. The host system of claim 1 , wherein the hardware processor further includes a register including a branch monitoring flag, and wherein the counter control unit is further configured to determine whether to perform the branch monitoring according to a value stored in the branch monitoring flag.
10. The host system of claim 1 , wherein the counter control unit is further configured to suspend the branch monitoring in response to a context switch, the context switch comprising the hardware processor switching from executing instructions of a first thread to executing instructions of a second thread.
11. The host system of claim 1 , wherein the counter control unit is further configured to suspend the branch monitoring in response to a hardware interrupt.
12. A method comprising:
employing a branch counter register of a hardware processor of a host system to store a count of branch instructions occurring within a sequence of instructions executed by the hardware processor;
employing an inter-branch instruction counter register of the hardware processor to store a count of instructions occurring between two consecutive branch instructions within the sequence of instructions;
employing a counter control unit of the hardware processor to perform branch monitoring, wherein branch monitoring comprises:
determining whether a selected instruction of the sequence of instructions is a branch instruction;
in response to determining whether the selected instruction is a branch instruction, when the selected instruction is a branch instruction, incrementing the value stored in the branch counter register; and
in response to determining whether the selected instruction is a branch instruction, when the selected instruction is not a branch instruction, incrementing the value stored in the inter-branch instruction counter register;
employing the counter control unit to trigger a switch event within the hardware processor according to a value stored in the branch counter register and according to a value stored in the inter-branch instruction counter register, wherein the switch event causes the hardware processor to switch from executing the sequence of instructions to executing an event handler routine;
in response to the switch event, employing the hardware processor to identify a target execution thread according to the sequence of instructions; and
in response to identifying the target execution thread, employing the hardware processor to determine whether the target execution thread is performing a code-reuse attack.
13. The method of claim 12 , further comprising:
employing the counter control unit to compare the value stored in the branch counter register to a predetermined threshold; and
in response, employing the counter control unit to determine whether to trigger the switch event according to a result of the comparison.
14. The method of claim 12 , further comprising:
employing the counter control unit to compare the value stored in the inter-branch instruction counter register to a predetermined threshold; and
in response, when the value stored in the inter-branch instruction counter register exceeds a predetermined threshold, resetting the value stored in the branch counter register to zero.
15. The method of claim 12 , wherein the switch event comprises an exception, and wherein the event handler routine comprises an exception handler registered in an interrupt descriptor table (IDT) of the processor.
16. The method of claim 12 , wherein the host system exposes a virtual machine, wherein the sequence of instructions consists of code executing within the virtual machine, and wherein the event handler routine executes outside the virtual machine.
17. The method of claim 12 , wherein branch instructions include at least one instruction selected from a group consisting of a jump instruction, a call instruction, and a return instruction.
18. The method of claim 12 , wherein the branch monitoring further comprises, when the selected instruction is a branch instruction, resetting the value stored in the inter-branch instruction counter register to zero.
19. The method of claim 12 , wherein the hardware processor further includes a register having a branch identifier field, and wherein the counter control unit is configured to determine whether the selected instruction is a branch instruction according to a value stored in the branch identifier field.
20. The method of claim 12 , wherein the hardware processor further includes a register including a branch monitoring flag, and wherein the counter control unit is further configured to determine whether to perform the branch monitoring according to a value stored in the branch monitoring flag.
21. The method of claim 12 , further comprising suspending the branch monitoring in response to a context switch, the context switch comprising the hardware processor switching from executing a first thread to executing a second thread.
22. The method of claim 12 , further comprising suspending the branch monitoring in response to a hardware interrupt.
23. A non-transitory computer-readable medium storing instructions which, when executed by a hardware processor of a host system, cause the host system to:
detect a switch event generated within the hardware processor while the hardware processor is executing an instruction sequence, the switch event causing the hardware processor to switch from executing the instruction sequence to executing an event handler routine;
in response to detecting the switch event, identify a target execution thread according to the instruction sequence; and
in response, determine whether the target execution thread is performing a code-reuse attack,
wherein the hardware processor comprises:
a branch counter register configured to store a count of branch instructions occurring within the instruction sequence;
an inter-branch instruction counter register configured to store a count of instructions occurring between two consecutive branch instructions within the instruction sequence; and
a counter control unit connected to the branch counter register, to the inter-branch instruction counter register, and to an instruction decoder module of the processor, the counter control unit comprising hardware logic configured to:
perform branch monitoring, and
trigger the switch event according to a value stored in the branch counter register and according to a value stored in the inter-branch instruction counter register,
wherein branch monitoring comprises:
determining whether a selected instruction of the sequence of instructions is a branch instruction;
in response to determining whether the selected instruction is a branch instruction, when the selected instruction is a branch instruction, incrementing the value stored in the branch counter register; and
in response to determining whether the selected instruction is a branch instruction, when the selected instruction is not a branch instruction, incrementing the value stored in the inter-branch instruction counter register.