Systems and methods for monitoring programs

1. A computer-implemented method for monitoring programs, at least a portion of the method being performed by a computing device comprising at least one processor, the method comprising:

placing a program within an enclave to perform a security analysis of the program to ascertain whether the program engages in malicious functionality, the enclave comprising a protected address space that code outside of the protected address space is restricted from accessing except through a special central processing unit enclave entry instruction;

hooking an application programming interface call within the program in the enclave to monitor the behavior of the program;

inserting the enclave entry instruction into code outside of the protected address space that the program accesses through the hooking of the application programming interface call; and

monitoring the behavior of the program by executing the program within the enclave in an attempt to force the program to use the hooked application programming interface call in order to access data outside the enclave, wherein:

executing the program comprises attempting, by the program, to bypass at least one instruction at an entry point of the application programming interface call; and

bypassing the instruction at the entry point of the application programming interface call further causes the program to bypass an enclave exit instruction and trigger an exception that enables revision of the program to perform further hooking and monitoring as part of the security analysis of the program to ascertain whether the program engages in malicious functionality.

2. The computer-implemented method of claim 1 , wherein the application programming interface call provides access to operating system resources.

3. The computer-implemented method of claim 2 , further comprising analyzing which specific application programming interface calls the program makes to identify what operating system resources the program is attempting to access.

4. The computer-implemented method of claim 1 , wherein hooking the application programming interface call comprises at least one of:

redirecting a function call;

modifying a function call; and

inserting an additional function call.

5. The computer-implemented method of claim 1 , wherein triggering the exception invokes an exception handler that enables hooking of the application programming interface call.

6. The computer-implemented method of claim 5 , wherein the exception handler replaces the application programming interface call with a call to another function that invokes another instance of the application programming interface call wrapped by code that hooks the application programming interface call.

7. The computer-implemented method of claim 1 , wherein monitoring the behavior of the program comprises altering the program during execution to enable further monitoring of application programming interface calls.

8. The computer-implemented method of claim 1 , wherein, prior to inserting the enclave entry instruction, the program already contains an enclave exit instruction.

9. The computer-implemented method of claim 8 , further comprising evaluating whether the program is digitally signed.

10. The computer-implemented method of claim 9 , further comprising lowering a safety score for the program based on a determination that the program already contains an enclave exit instruction and a determination that the program is not digitally signed.

11. The computer-implemented method of claim 1 , further comprising performing static analysis to identify application programming interface calls within the program prior to executing the program within the enclave.

12. A system for monitoring programs, the system comprising:

a placement module, stored in memory, that places a program within an enclave to perform a security analysis of the program to ascertain whether the program engages in malicious functionality, the enclave comprising a protected address space that code outside of the protected address space is restricted from accessing except through a special central processing unit enclave entry instruction;

a hooking module, stored in memory, that hooks an application programming interface call within the program in the enclave to monitor the behavior of the program;

an insertion module, stored in memory, that inserts the enclave entry instruction into code outside of the protected address space that the program accesses through the hooking of the application programming interface call;

a monitoring module, stored in memory, that monitors the behavior of the program by executing the program within the enclave in an attempt to force the program to use the hooked application programming interface call in order to access data outside the enclave; and

at least one physical processor configured to execute the placement module, the hooking module, the insertion module, and the monitoring module, wherein:

the monitoring module detects whether the program attempts to bypass at least one instruction at an entry point of the application programming interface call; and

the monitoring module detects whether the program bypasses an enclave exit instruction and triggers an exception that enables revision of the program to perform further hooking and monitoring as part of the security analysis of the program to ascertain whether the program engages in malicious functionality.

13. The system of claim 12 , wherein the application programming interface call provides access to operating system resources.

14. The system of claim 13 , wherein the monitoring module analyzes which specific application programming interface calls the program makes to identify what operating system resources the program is attempting to access.

15. The system of claim 12 , wherein the hooking module hooks the application programming interface call by performing at least one of:

redirecting a function call;

modifying a function call; and

inserting an additional function call.

16. The system of claim 12 , wherein triggering the exception invokes an exception handler that enables hooking of the application programming interface call.

17. The system of claim 16 , wherein the exception handler is configured to replace the application programming interface call with a call to another function that invokes another instance of the application programming interface call wrapped by code that hooks the application programming interface call.

18. The system of claim 12 , wherein the monitoring module monitors the behavior of the program by altering the program during execution to enable further monitoring of application programming interface calls.

19. The system of claim 12 , wherein the insertion module is configured to determine whether the program already contains an enclave exit instruction prior to inserting the enclave entry instruction.

20. A non-transitory computer-readable medium comprising one or more computer-readable instructions that, when executed by at least one processor of a computing device, cause the computing device to:

place a program within an enclave to perform a security analysis of the program to ascertain whether the program engages in malicious functionality, the enclave comprising a protected address space that code outside of the protected address space is restricted from accessing except through a special central processing unit enclave entry instruction;

hook an application programming interface call within the program in the enclave to monitor the behavior of the program;

insert the enclave entry instruction into code outside of the protected address space that the program accesses through the hooking of the application programming interface call; and

monitor the behavior of the program by executing the program within the enclave in an attempt to force the program to use the hooked application programming interface call in order to access data outside the enclave, wherein:

executing the program comprises attempting, by the program, to bypass at least one instruction at an entry point of the application programming interface call; and

bypassing the instruction at the entry point of the application programming interface call further causes the program to bypass an enclave exit instruction and trigger an exception that enables revision of the program to perform further hooking and monitoring as part of the security analysis of the program to ascertain whether the program engages in malicious functionality.