System and method for reverse command shell detection

1. A method for tracking a reverse command shell intrusion on a user device, the method comprising:

detecting a process starting on the user device;

preparing the process to monitor Application Programming Interface (API) calls between the process and an operating system of the user device;

determining whether the process is associated with a reverse command shell intrusion based on information associated with the process and/or the API calls by:

detecting API calls that are associated with creating a pipe;

tracking file handles of the pipe; and

determining a possible existence of a reverse shell intrusion upon a child process of the process in response to detecting a first API call and determining that the first API call is writing to the child process; and

executing security policies against the process associated with the reverse command shell intrusion.

2. The method of claim 1 , wherein determining whether the process is associated with a reverse command shell intrusion based upon the information associated with the process and/or the API calls comprises:

detecting API calls that are associated with network connections; and

determining a possible existence of a reverse shell intrusion upon a child process of the process in response to determining that, based on the API calls, the parent process is receiving requests to open a connection to a network application located in a network that is remote and/or that the parent process is receiving data over a connection to a network application located in a network that is remote.

3. The method of claim 1 , wherein determining that the first API call is writing to the child process comprises determining that at least one file handle of the pipe is attached to a stdin stream of the child process.

4. The method of claim 1 , further comprising;

examining read and/or write operations upon at least one file handle of the pipe;

identifying commands passed by the parent process to the child process via the at least one file handle of the pipe; and

concluding whether the child process is associated with a reverse command shell intrusion based on the commands.

5. The method of claim 1 , wherein determining whether the process is associated with a reverse command shell intrusion based upon the information associated with the process and/or the API calls comprises:

detecting API calls that are associated with creating a pipe;

tracking file handles of the pipe;

detecting API calls that are associated with creating a new child process of the process, the process being a parent process of the new child process; and

determining a possible existence of a reverse shell intrusion upon the new child process in response to determining that the new child process is a command shell process or an interpreter process.

6. The method of claim 5 , further comprising:

identifying process startup information of the new child process; and

determining a possible existence of a reverse shell intrusion upon the new child process in response to determining that a stdin stream and a stdout stream of the new child process are redirected to the file handles of the pipe.

7. The method of claim 5 , further comprising:

identifying that a stdin stream and a stdout stream of the new child process are redirected to the file handles of the pipe;

examining read and/or write operations upon at least one file handle of the pipe;

identifying commands passed by the parent process to the new child process via the at least one file handle of the pipe; and

concluding whether the new child process is associated with a reverse command shell intrusion based on the commands.

8. The method of claim 5 , further comprising:

examining read and/or write operations upon the file handles of the pipe;

identifying output sent from the new child process to the parent process via the file handles of the pipe; and

concluding whether the new child process is associated with a reverse command shell intrusion by determining whether the new child process sent its output in response to the commands.

9. A system for tracking a reverse command shell intrusion on a user device, the system comprising:

an endpoint telemetry aggregation system, executed by a processor, that stores security policies for remediating processes executing on the user device that are associated with a reverse command shell intrusion; and

a security agent executing by a processor on the user device that:

prepares a process to monitor Application Programming Interface (API) calls between the process and an operating system of the user device;

determines whether the process is associated with a reverse command shell intrusion based on information associated with the process and/or the API calls by:

detecting API calls that are associated with creating a pipe;

tracking file handles of the pipe; and

determining a possible existence of a reverse shell intrusion upon a child process of the process in response to detecting a first API call and determining that the first API call is writing to the child process; and

requests the security policies from the endpoint telemetry aggregation system; and

executes the security policies against the process associated with the reverse command shell intrusion.

10. The system of claim 9 , wherein the security agent determines whether the process is associated with a reverse command shell intrusion by detecting API calls that are associated with network connections, and determining a possible existence of a reverse shell intrusion upon a child process of the process in response to determining that, based on the API calls, the parent process is receiving requests to open a connection to a network application located in a network that is remote and/or that the parent process is receiving data over a connection to a network application located in a network that is remote.

11. The system of claim 9 , wherein determining that the first API call is writing to the child process comprises determining that at least one file handle of the pipe is attached to a stdin stream of the child process.

12. The system of claim 9 , wherein the security agent examines read and/or write operations upon at least one file handle of the pipe, identifies commands passed by the parent process to the child process via the at least one file handle of the pipe; and concludes whether the child process is associated with a reverse command shell intrusion based on the commands.

13. The system of claim 9 , wherein the security agent determines whether the process is associated with a reverse command shell intrusion by detecting API calls that are associated with creating a pipe, tracking file handles of the pipe, detecting API calls that are associated with creating a new child process of the process, the process being a parent process of the new child process, and determining a possible existence of a reverse shell intrusion upon the new child process in response to determining that the new child process is a command shell process or an interpreter process.

14. The system of claim 13 , wherein the security agent identifies process startup information of the new child process and determines a possible existence of a reverse shell intrusion upon the new child process in response to determining that a stdin stream and a stdout stream of the new child process are redirected to the file handles of the pipe.

15. The system of claim 13 , wherein the security agent identifies that a stdin stream and a stdout stream of the new child process are redirected to the file handles of the pipe, examines read and/or write operations upon at least one file handle of the pipe, identifies commands passed by the parent process to the new child process via the at least one file handle of the pipe, and concludes whether the new child process is associated with a reverse command shell intrusion based on the commands.

16. The system of claim 13 , wherein the security agent examines read and/or write operations upon the file handles of the pipe, identifies output sent from the new child process to the parent process via the file handles of the pipe, and concludes whether the new child process is associated with a reverse command shell intrusion by determining whether the new child process sent its output in response to the commands.

17. The system of claim 9 , wherein the process associated with the reverse command shell intrusion is a command-line interpreter program that is native to the operating system of the user device, a script interpreter program, or a database application that takes Software Query Language (SQL) commands as input for accessing a database.

18. The method of claim 1 , further comprising determining whether the process is associated with a reverse command shell intrusion by determining that the process is a command shell process and analyzing a parent process of the command shell process, detecting the first API call, determining that a file descriptor of the first API call matches a handle to a stdin stream of the command shell process, identifying commands passed by the parent process to the command shell process in the stdin stream of the command shell process, and concluding whether the process is a reverse command shell intrusion based on the commands.

19. The system of claim 9 , wherein the security agent further determines whether the process is associated with a reverse command shell intrusion by determining that the process is a command shell process and analyzing a parent process of the command shell process, detecting a first API call, determining that a file descriptor of the first API call matches a handle to a stdin stream of the command shell process, identifying commands passed by the parent process to the command shell process in the stdin stream of the command shell process, and concluding whether the process is a reverse command shell intrusion based on the commands.