Detecting return-oriented programming payloads by evaluating data for a gadget address space address and determining whether operations associated with instructions beginning at the address indicate a return-oriented programming payload
View Patent ↗Systems, methods, and media for detecting the presence of return-oriented programming (ROP) payloads are provided, comprising: identifying a potential gadget address space; determining if a piece of the data corresponds to an address of the potential gadget address space; and in response to determining that the piece of the data corresponds to an address of the potential gadget address space: determining whether a plurality of operations, each associated one of a plurality instructions beginning at the address, indicates that an ROP payload is present in the data, and indicating that an ROP payload is present in the data in response to making a determination that a plurality of operations indicates that an ROP payload is present in the data a given number of times.
1. A system for detecting the presence of a return-oriented programming (ROP) payload in data, comprising:
a memory; and
a hardware processor that:
for each instruction of a plurality of instructions beginning at an address corresponding to a piece of data in the data:
attempts to execute the instruction;
counts the instruction as part of an instruction count;
at least one of: determines that the instruction does not have an invalid execution address; determines that the instruction is not invalid; and determines that the instruction is not privileged; and
determines that the instruction count is not greater than or equal to at least one threshold;
in response to:
the at least one of: determining that the instruction does not have an invalid execution address; determining that the instruction is not invalid; and determining that the instruction is not privileged; and
the determining that the instruction count is not greater than or equal to the at least one threshold,
increases a gadget count; and
indicates that an ROP payload is present in the data in response to the gadget count meeting a threshold greater than one.
2. The system of claim 1 , wherein in attempting to execute the instruction, the hardware processor attempts to execute the instruction using an emulator.
3. The system of claim 1 , wherein in the at least one of: determining that the instruction does not have an invalid execution address; determining that the instruction is not invalid; and determining that the instruction is not privileged, the hardware processor determines that the instruction does not have an invalid execution address.
4. The system of claim 1 , wherein in the at least one of: determining that the instruction does not have an invalid execution address; determining that the instruction is not invalid; and determining that the instruction is not privileged, the hardware processor determines that the instruction is not a privileged instruction.
5. The system of claim 1 , wherein increasing the gadget count is in response to the hardware processor also determining that an indirect control transfer instruction that is controlled by an unchanged piece of the data has been effected.
6. The system of claim 1 , wherein increasing the gadget count is in response to the hardware processor also determining that a return instruction that is controlled by an unchanged piece of the data has been effected.
7. The system of claim 1 , wherein the instruction count is a count of the number of instructions in a gadget.
8. The system of claim 1 , wherein the instruction count is a count of the total number of instructions executed.
9. A method for detecting the presence of a return-oriented programming (ROP) payload in data, comprising:
for each instruction of a plurality of instructions beginning at an address corresponding to a piece of data in the data, using a hardware processor to:
attempt to execute the instruction;
count the instruction as part of an instruction count;
at least one of: determine that the instruction does not have an invalid execution address; determine that the instruction is not invalid; and determine that the instruction is not privileged; and
determine that the instruction count is not greater than or equal to at least one threshold;
in response to:
the at least one of: determining that the instruction does not have an invalid execution address; determining that the instruction is not invalid; and determining that the instruction is not privileged; and
the determining that the instruction count is not greater than or equal to the at least one threshold,
using the hardware processor to increase a gadget count; and
indicating, using the hardware processor, that an ROP payload is present in the data in response to the gadget count meeting a threshold greater than one.
10. The method of claim 9 , wherein attempting to execute the instruction comprises attempting to execute the instruction using an emulator.
11. The method of claim 9 , wherein the at least one of: determining that the instruction does not have an invalid execution address; determining that the instruction is not invalid; and determining that the instruction is not privileged comprises determining that the instruction does not have an invalid execution address.
12. The method of claim 9 , wherein the at least one of: determining that the instruction does not have an invalid execution address; determining that the instruction is not invalid; and determining that the instruction is not privileged comprises determining that the instruction is not a privileged instruction.
13. The method of claim 9 , wherein increasing the gadget count is also in response to determining that an indirect control transfer instruction that is controlled by an unchanged piece of the data has been effected.
14. The method of claim 9 , wherein increasing the gadget count is also in response to determining that a return instruction that is controlled by an unchanged piece of the data has been effected.
15. The method of claim 9 , wherein the instruction count is a count of the number of instructions in a gadget.
16. The method of claim 9 , wherein the instruction count is a count of the total number of instructions executed.
17. A non-transitory computer-readable medium containing computer-executable instructions that, when executed by a processor, cause the processor to perform a method for detecting the presence of a return-oriented programming (ROP) payload in data, the method comprising:
for each instruction of a plurality of instructions beginning at an address corresponding to a piece of data in the data:
attempting to execute the instruction;
counting the instruction as part of an instruction count;
at least one of: determining that the instruction does not have an invalid execution address; determining that the instruction is not invalid; and determining that the instruction is not privileged; and
determining that the instruction count is not greater than or equal to at least one threshold;
in response to:
the at least one of: determining that the instruction does not have an invalid execution address; determining that the instruction is not invalid; and determining that the instruction is not privileged; and
the determining that the instruction count is not greater than or equal to the at least one threshold,
increasing a gadget count; and
indicating that an ROP payload is present in the data in response to the gadget count meeting a threshold greater than one.
18. The non-transitory computer-readable medium of claim 17 , wherein attempting to execute the instruction comprises attempting to execute the instruction using an emulator.
19. The non-transitory computer-readable medium of claim 17 , wherein the at least one of: determining that the instruction does not have an invalid execution address; determining that the instruction is not invalid; and determining that the instruction is not privileged comprises determining that the instruction does not have an invalid execution address.
20. The non-transitory computer-readable medium of claim 17 , wherein the at least one of: determining that the instruction does not have an invalid execution address; determining that the instruction is not invalid; and determining that the instruction is not privileged comprises determining that the instruction is not a privileged instruction.