Method and apparatus for detecting shellcode
View Patent ↗The invention is a method and apparatus for detecting shellcode such that a set of computer instructions is scanned for the presence of a null operation instruction. The computer instructions are also examined for the presence of a system call instruction, and reviewed for the presence of a decoder instruction set. A null operation weight value is then determined corresponding to the null operation instruction. Also assessed is a system call weight value corresponding to the system call instruction. In addition, a decoder weight value is calculated corresponding to the decoder instruction set. The null operation weight value, the system call weight value, and the decoder weight value are then analyzed to identify a shellcode.
1. A computer-implemented method, comprising:
scanning a set of computer instructions for a null operation instruction; and
detecting predetermined code based at least in part on the null operation instruction;
wherein the predetermined code is detected based at least in part on a decoder instruction set including a jump instruction for acting as an encoded system call instruction, the jump instruction invoking a function in an absence of a system call instruction from the set of computer instructions.
2. The method of claim 1 , wherein the predetermined code is detected based at least in part on another system call instruction.
3. A computer-implemented method, comprising:
scanning a set of computer instructions for a null operation instruction;
reviewing the set of computer instructions for the presence of a decoder instruction set including a jump instruction for acting as an encoded system call instruction, the jump instruction invoking a function in the absence of a system call instruction from the set of computer instructions; and
detecting predetermined code based at least in part on the null operation instruction;
wherein the predetermined code includes shellcode.
4. A computer-implemented method, comprising:
scanning a set of computer instructions for a system call instruction; and
detecting predetermined code based at least in part on the system call instruction;
wherein the predetermined code is detected based at least in part on a decoder instruction set including a jump instruction for acting as an encoded system call instruction, the jump instruction invoking a function in an absence of the system call instruction from the set of computer instructions.
5. The method of claim 4 , wherein the predetermined code is detected based at least in part on a null operation instruction.
6. A computer-implemented method, comprising:
scanning a set of computer instructions for a system call instruction;
reviewing the set of computer instructions for the presence of a decoder instruction set including a jump instruction for acting as an encoded system call instruction, the jump instruction invoking a function in the absence of the system call instruction from the set of computer instructions; and
detecting predetermined code based at least in part on the system call instruction;
wherein the predetermined code includes shellcode.
7. A computer program product embodied on a non-transitory computer readable medium, comprising:
computer code for scanning a set of computer instructions for a decoder instruction set; and
computer code for detecting predetermined code based at least in part on the decoder instruction set including a jump instruction for acting as an encoded system call instruction, the jump instruction invoking a function in an absence of a system call instruction from the set of computer instructions.
8. The computer program product of claim 7 , wherein the predetermined code is detected based at least in part on another system call instruction.
9. The computer program product of claim 7 , wherein the predetermined code is detected based at least in part on a null operation instruction.
10. The computer program product of claim 7 , wherein the predetermined code includes shellcode.
11. A method of detecting a shellcode, comprising:
scanning a set of computer instructions for at least one of a null operation instruction, a system call instruction, and a decoder instruction set; and
detecting a shellcode based at least in part on at least one of the null operation instruction, the system call instruction, and the decoder instruction set;
wherein the decoder instruction set includes a jump instruction for acting as an encoded system call instruction, the jump instruction invoking a function in an absence of the system call instruction from the set of computer instructions.
12. A computer program product embodied on a non-transitory computer readable medium, comprising:
computer code for scanning a set of computer instructions for a null operation instruction; and
computer code for detecting predetermined code based at least in part on the null operation instruction;
wherein the computer program product is operable such that the predetermined code is detected based at least in part on a decoder instruction set including a jump instruction for acting as an encoded system call instruction, the jump instruction invoking a function in an absence of a system call instruction from the set of computer instructions.
13. A system, comprising:
a processor for:
scanning a set of computer instructions for a null operation instruction; and
detecting predetermined code based at least in part on the null operation instruction;
wherein the system is operable such that the predetermined code is detected based at least in part on a decoder instruction set including a jump instruction for acting as an encoded system call instruction, the jump instruction invoking a function in an absence of a system call instruction from the set of computer instructions.
14. A computer program product embodied on a non-transitory computer readable medium, comprising:
computer code for scanning a set of computer instructions for a null operation instruction;
computer code for reviewing the set of computer instructions for the presence of a decoder instruction set including a jump instruction for acting as an encoded system call instruction, the jump instruction invoking a function in the absence of a system call instruction from the set of computer instructions; and
computer code for detecting predetermined code based at least in part on the null operation instruction;
wherein the predetermined code includes shellcode.
15. A system, comprising:
a processor for:
scanning a set of computer instructions for a null operation instruction;
reviewing the set of computer instructions for the presence of a decoder instruction set including a jump instruction for acting as an encoded system call instruction, the jump instruction invoking a function in the absence of a system call instruction from the set of computer instructions; and
detecting predetermined code based at least in part on the null operation instruction;
wherein the predetermined code includes shellcode.
16. A computer program product embodied on a non-transitory computer readable medium, comprising:
computer code for scanning a set of computer instructions for a system call instruction; and
computer code for detecting predetermined code based at least in part on the system call instruction;
wherein the computer program product is operable such that the predetermined code is detected based at least in part on a decoder instruction set including a jump instruction for acting as an encoded system call instruction, the jump instruction invoking a function in an absence of the system call instruction from the set of computer instructions.
17. A system, comprising:
a processor for:
scanning a set of computer instructions for a system call instruction; and
detecting predetermined code based at least in part on the system call instruction;
wherein the system is operable such that the predetermined code is detected based at least in part on a decoder instruction set including a jump instruction for acting as an encoded system call instruction, the jump instruction invoking a function in an absence of the system call instruction from the set of computer instructions.
18. A computer program product embodied on a non-transitory computer readable medium, comprising:
computer code for scanning a set of computer instructions for a system call instruction;
computer code for reviewing the set of computer instructions for the presence of a decoder instruction set including a jump instruction for acting as an encoded system call instruction, the jump instruction invoking a function in the absence of the system call instruction from the set of computer instructions; and
computer code for detecting predetermined code based at least in part on the system call instruction;
wherein the predetermined code includes shellcode.
19. A system, comprising:
a processor for:
scanning a set of computer instructions for a system call instruction;
reviewing the set of computer instructions for the presence of a decoder instruction set including a jump instruction for acting as an encoded system call instruction, the jump instruction invoking a function in the absence of the system call instruction from the set of computer instructions; and
detecting predetermined code based at least in part on the system call instruction;
wherein the predetermined code includes shellcode.
20. A method, comprising:
scanning a set of computer instructions for a decoder instruction set; and
detecting predetermined code based at least in part on the decoder instruction set including a jump instruction for acting as an encoded system call instruction, the jump instruction invoking a function in an absence of a system call instruction from the set of computer instructions.
21. A system, comprising:
a processor for:
scanning a set of computer instructions for a decoder instruction set; and
detecting predetermined code based at least in part on the decoder instruction set including a jump instruction for acting as an encoded system call instruction, the jump instruction invoking a function in an absence of a system call instruction from the set of computer instructions.
22. A computer program product embodied on a non-transitory computer readable medium, comprising:
computer code for scanning a set of computer instructions for at least one of a null operation instruction, a system call instruction, and a decoder instruction set; and
computer code for detecting a shellcode based at least in part on at least one of the null operation instruction, the system call instruction, and the decoder instruction set;
wherein the decoder instruction set includes a jump instruction for acting as an encoded system call instruction, the jump instruction invoking a function in an absence of the system call instruction from the set of computer instructions.
23. A system, comprising:
a processor for:
scanning a set of computer instructions for at least one of a null operation instruction, a system call instruction, and a decoder instruction set; and
detecting a shellcode based at least in part on at least one of the null operation instruction, the system call instruction, and the decoder instruction set;
wherein the decoder instruction set includes a jump instruction for acting as an encoded system call instruction, the jump instruction invoking a function in an absence of the system call instruction from the set of computer instructions.
24. The method of claim 1 , further comprising establishing a null operation instruction threshold that reflects a number of times the null operation instruction occurs in an input string.
25. The method of claim 3 , further comprising establishing threshold values for characteristics of the shellcode, and calculating weight values based on the characteristics including a minimum instruction threshold and a maximum instruction threshold.