Automated and scalable worker orchestration for cloud-based computer forensic analysis
Disclosed are techniques for performing forensic analysis of computer systems in a cloud network. The techniques can include using a scalable, cloud-based, specialized computer architecture for performing the forensic analysis of computer systems.
1. A computer-implemented method, comprising:
collecting a plurality of items of forensic evidence from an attacked network, wherein the attacked network was previously subject to a security incident;
generating a set of tasks based on the plurality of items of the forensic evidence collected from the attacked network, wherein each task of the set of tasks is associated with an item of the forensic evidence of the plurality of items of the forensic evidence, and wherein the set of tasks is stored in a task queue;
determining a number of workers to generate based on the set of tasks stored in the task queue, wherein
each worker of the number of workers is configured to process a respective task of the set of tasks, and
determining each of the number of workers comprises:
detecting the respective task among the set of tasks stored in the task queue; and
analyzing the respective task to determine a condition for execution of the respective task, wherein the condition for the execution of the respective task is related to requirement of acquiring a new item of the forensic evidence from the attacked network;
generating the number of workers in a cloud network based on the analyzing of the respective task; and
executing the number of workers in parallel to process the set of tasks stored in the task queue.
2. The computer-implemented method of claim 1 , wherein determining the number of workers to generate further comprises:
detecting a particular task from amongst the set of tasks stored in the task queue;
determining that the particular task includes acquiring the new item of the forensic evidence from the attacked network; and
generating a new worker of the number of workers to process the particular task.
3. The computer-implemented method of claim 1 , wherein determining the number of workers to generate further comprises:
detecting a particular task from amongst the set of tasks stored in the task queue, the particular task being associated with a particular item of the forensic evidence of the plurality of items of the forensic evidence;
determining that the particular task does not include acquiring the new item of the forensic evidence from the attacked network;
identifying a previously-generated worker associated with the particular item of the forensic evidence; and
processing the particular task using the previously-generated worker and the particular item of the forensic evidence.
4. The computer-implemented method of claim 1 , wherein executing the number of workers in parallel includes:
detecting a particular task from the set of tasks stored in the task queue,
retrieving a particular item of the forensic evidence associated with the particular task, and
performing the particular task using the particular item of the forensic evidence associated with the particular task.
5. The computer-implemented method of claim 1 , wherein executing the number of workers in parallel further comprises:
assigning a particular item of the forensic evidence of the plurality of items of the forensic evidence to a particular worker of the number of workers;
determining that a particular task corresponds to the particular item of the forensic evidence;
determining that a particular worker of the number of workers is configured to process the particular task using the particular item of the forensic evidence, or determining that a main server is configured to process the particular task using the particular item of the forensic evidence; and
processing the particular task using the particular worker or the main server, depending on whether the particular task was processed by the particular worker or the main server.
6. The computer-implemented method of claim 1 , further comprising:
determining that a particular task of the set of tasks has one or more dependent tasks, wherein each dependent task of the one or more dependent tasks is processed in association with the particular task;
processing the particular task and the one or more dependent tasks;
detecting malicious activity within the attacked network; and
in response to detecting the malicious activity, adding one or more additional tasks to the set of tasks included in the task queue.
7. The computer-implemented method of claim 1 , further comprising:
determining that the set of tasks has been processed by the number of workers; and
in response to determining that the set of tasks has been processed, terminating each worker of the number of workers, wherein terminating each worker of the number of workers reduces a processing burden imposed on cloud compute resources.
8. A non-transitory computer-readable medium comprising instructions that are executable by a processing device for causing the processing device to perform operations comprising:
collecting a plurality of items of forensic evidence from an attacked network, wherein the attacked network was previously subject to a security incident;
generating a set of tasks based on the plurality of items of the forensic evidence collected from the attacked network, wherein each task of the set of tasks is associated with an item of the forensic evidence of the plurality of items of the forensic evidence, and wherein the set of tasks is stored in a task queue;
determining a number of workers to generate based on the set of tasks stored in the task queue, wherein
each worker of the number of workers is configured to process a respective task of the set of tasks, and
determining each of the number of workers comprises:
detecting the respective task among the set of tasks stored in the task queue; and
analyzing the respective task to determine a condition for execution of the respective task, wherein the condition for the execution of the respective task is related to requirement of acquiring a new item of the forensic evidence from the attacked network;
generating the number of workers in a cloud network based on the analyzing of the respective task; and
executing the number of workers in parallel to process the set of tasks stored in the task queue.
9. The non-transitory computer-readable medium of claim 8 , wherein determining the number of workers to generate further comprises:
detecting a particular task from amongst the set of tasks stored in the task queue;
determining that the particular task includes acquiring the new item of the plurality of items of the forensic evidence from the attacked network; and
generating a new worker of the number of workers to process the particular task.
10. The non-transitory computer-readable medium of claim 8 , wherein determining the number of workers to generate further comprises:
detecting a particular task from amongst the set of tasks stored in the task queue, the particular task being associated with a particular item of the forensic evidence of the plurality of items of the forensic evidence;
determining that the particular task does not include acquiring the new item of the forensic evidence from the attacked network;
identifying a previously-generated worker associated with the particular item of the forensic evidence; and
processing the particular task using the previously-generated worker and the particular item of the forensic evidence.
11. The non-transitory computer-readable medium of claim 8 , wherein executing the number of workers in parallel includes:
detecting a particular task from the set of tasks stored in the task queue,
retrieving a particular item of the forensic evidence associated with the particular task, and
performing the particular task using the particular item of the forensic evidence associated with the particular task.
12. The non-transitory computer-readable medium of claim 8 , wherein executing the number of workers in parallel further comprises:
assigning a particular item of the forensic evidence of the plurality of items of the forensic evidence to a particular worker of the number of workers;
determining that a particular task corresponds to the particular item of the forensic evidence;
determining that a particular worker of the number of workers is configured to process the particular task using the particular item of the forensic evidence, or determining that a main server is configured to process the particular task using the particular item of the forensic evidence; and
processing the particular task using the particular worker or the main server, depending on whether the particular task was processed by the particular worker or the main server.
13. The non-transitory computer-readable medium of claim 8 , wherein the operations further comprise:
determining that a particular task of the set of tasks has one or more dependent tasks, wherein each dependent task of the one or more dependent tasks is processed in association with the particular task;
processing the particular task and the one or more dependent tasks;
detecting malicious activity within the attacked network; and
in response to detecting the malicious activity, adding one or more additional tasks to the set of tasks included in the task queue.
14. The non-transitory computer-readable medium of claim 8 , wherein the operations further comprise:
determining that the set of tasks has been processed by the number of workers; and
in response to determining that the set of tasks has been processed, terminating each worker of the number of workers, wherein terminating each worker of the number of workers reduces a processing burden imposed on cloud compute resources.
15. A system, comprising:
one or more processors; and
a non-transitory computer-readable storage medium containing instructions which, when executed on the one or more processors, cause the one or more processors to perform operations including:
collecting a plurality of items of forensic evidence from an attacked network, wherein the attacked network was previously subject to a security incident;
generating a set of tasks based on the plurality of items of the forensic evidence collected from the attacked network, wherein each task of the set of tasks is associated with an item of the forensic evidence of the plurality of items of the forensic evidence, and wherein the set of tasks is stored in a task queue;
determining a number of workers to generate based on the set of tasks stored in the task queue, wherein
each worker of the number of workers is configured to process a respective task of the set of tasks, and
determining each of the number of workers comprises:
detecting the respective task among the set of tasks stored in the task queue; and
analyzing the respective task to determine a condition for execution of the respective task, wherein the condition for the execution of the respective task is related to requirement of acquiring a new item of the forensic evidence from the attacked network;
generating the number of workers in a cloud network based on the analyzing of the respective task; and
executing the number of workers in parallel to process the set of tasks stored in the task queue.
16. The system of claim 15 , wherein determining the number of workers to generate further comprises:
detecting a particular task from amongst the set of tasks stored in the task queue;
determining that the particular task includes acquiring the new item of the plurality of items of the forensic evidence from the attacked network; and
generating a new worker of the number of workers to process the particular task.
17. The system of claim 15 , wherein determining the number of workers to generate further comprises:
detecting a particular task from amongst the set of tasks stored in the task queue, the particular task being associated with a particular item of the forensic evidence of the plurality of items of the forensic evidence;
determining that the particular task does not include acquiring the new item of the forensic evidence from the attacked network;
identifying a previously-generated worker associated with the particular item of the forensic evidence; and
processing the particular task using the previously-generated worker and the particular item of the forensic evidence.
18. The system of claim 15 , wherein executing the number of workers in parallel includes:
detecting a particular task from the set of tasks stored in the task queue,
retrieving a particular item of the forensic evidence associated with the particular task, and
performing the particular task using the particular item of the forensic evidence associated with the particular task.
19. The system of claim 15 , wherein executing the number of workers in parallel further comprises:
assigning a particular item of the forensic evidence of the plurality of items of the forensic evidence to a particular worker of the number of workers;
determining that a particular task corresponds to the particular item of the forensic evidence;
determining that a particular worker of the number of workers is configured to process the particular task using the particular item of the forensic evidence, or determining that a main server is configured to process the particular task using the particular item of the forensic evidence; and
processing the particular task using the particular worker or the main server, depending on whether the particular task was processed by the particular worker or the main server.
20. The system of claim 15 , wherein the operations further comprise:
determining that a particular task of the set of tasks has one or more dependent tasks, wherein each dependent task of the one or more dependent tasks is processed in association with the particular task;
processing the particular task and the one or more dependent tasks;
detecting malicious activity within the attacked network; and
in response to detecting the malicious activity, adding one or more additional tasks to the set of tasks included in the task queue.