Real-time detection of system threats
Some examples relate generally to managing and storing data, and more specifically to the real-time detection of ransomware, system (or insider) threats, or the misappropriation of credentials by using file system audit events.
1. A method for detection of a threat in a file system, the method comprising:
accessing audit events in the file system for a time interval, the audit events including unique file operations and duplicative file operations within the time interval;
de-duplicating the audit events to remove the duplicative file operations and retain the unique file operations from the audit events;
generating time series data that comprises the unique file operations and is devoid of the duplicative file operations;
analyzing the time series data to determine whether a subset of the unique file operations includes file-read instructions to copy files corresponding to the subset of unique file operations;
comparing a pattern of the file-read instructions in the time interval to a normal pattern of file-read instructions;
determining, based at least in part on the comparing, that the file-read instructions in the subset of unique file operations are abnormal based at least in part on a deviation between the pattern of the file-read instructions in the time interval and the normal pattern of file-read instructions;
responsive to determining that the file-read instructions in the subset of unique file operations are abnormal, determining that the file system is subject to a threat; and
generating an alert based at least in part on determining that the file system is subject to the threat.
2. The method of claim 1 , wherein the audit events include information comprising, for each audit event, a user id, a file name, a type of access, and a timestamp.
3. The method of claim 1 , wherein de-duplicating the audit events is based at least in part on an identification of successive file operations that do not lead to a change in a file state.
4. The method of claim 1 , further comprising:
generating a finite state machine including one or more file states, the file states including a file open state, a file read state, a file write state, a file read/write state, and a file close state; and
storing the file states in the finite state machine in a key-value object store.
5. The method of claim 1 , wherein determining that the file-read instructions in the subset of unique file operations are abnormal comprises applying a set of machine learning models to the audit events, the set of machine learning models trained to determine the pattern or a number of the file-read instructions and to compare the pattern or the number of the file-read instructions to the normal pattern of file-read instructions or a normal number of file-read instructions based on features representing a normal or expected behavior of the file system.
6. The method of claim 4 , wherein de-duplicating the audit events includes maintaining a file system state based on the finite state machine.
7. The method of claim 1 , wherein determining that the file-read instructions in the subset of unique file operations are abnormal comprises applying Seasonal-Trend Decomposition Procedure Based on Loess (STL) decomposition to file delete audit events to remove seasonal and trend components and using a residue of the decomposition to generate the time series data, and performing an Exploratory Data Analysis (ESD) test on the time series data.
8. A system for detection of a threat in a file system, the system comprising:
at least one processor for executing machine-readable instructions; and
a memory storing instructions configured to cause the at least one processor to perform operations comprising, at least:
accessing audit events in the file system for a time interval, the audit events including unique file operations and duplicative file operations within the time interval;
de-duplicating the audit events to remove the duplicative file operations and retain the unique file operations from the audit events;
generating time series data that comprises the unique file operations and is devoid of the duplicative file operations;
analyzing the time series data to determine whether a subset of the unique file operations includes file-read instructions to copy files corresponding to the subset of unique file operations;
comparing a pattern of the file-read instructions in the time interval to a normal pattern of file-read instructions;
determining, based at least in part on the comparing, that the file-read instructions in the subset of unique file operations are abnormal based at least in part on a deviation between the pattern of the file-read instructions in the time interval and the normal pattern of file-read instructions;
responsive to determining that the file-read instructions in the subset of unique file operations are abnormal, determining that the file system is subject to a threat; and
generating an alert based at least in part on determining that the file system is subject to the threat.
9. The system of claim 8 , wherein the audit events include information comprising, for each audit event, a user id, a file name, a type of access, and a timestamp.
10. The system of claim 8 , wherein de-duplicating the audit events is based at least in part on an identification of successive file operations that do not lead to a change in a file state.
11. The system of claim 8 , wherein the operations further comprise:
generating a finite state machine including one or more file states, the file states including a file open state, a file read state, a file write state, a file read/write state, and a file close state; and
storing the file states in the finite state machine in a key-value object store.
12. The system of claim 8 , wherein determining that the file-read instructions in the subset of unique file operations are abnormal comprises applying a set of machine learning models to the audit events, the set of machine learning models trained to determine the pattern or a number of the file-read instructions and to compare the pattern or the number of the file-read instructions to the normal pattern of file-read instructions or a normal number of file-read instructions based on features representing a normal or expected behavior of the file system.
13. The system of claim 11 , wherein de-duplicating the audit events includes maintaining a file system state based on the finite state machine.
14. The system of claim 8 , wherein determining that the file-read instructions in the subset of unique file operations are abnormal comprises applying Seasonal-Trend Decomposition Procedure Based on Loess (STL) decomposition to file delete audit events to remove seasonal and trend components and using a residue of the decomposition to generate the time series data, and performing an Exploratory Data Analysis (ESD) test on the time series data.
15. A non-transitory, machine-readable medium storing instructions which, when read by a machine, cause the machine to perform operations comprising, at least:
accessing audit events in a file system for a time interval, the audit events including unique file operations and duplicative file operations within the time interval;
de-duplicating the audit events to remove the duplicative file operations and retain the unique file operations from the audit events;
generating time series data that comprises the unique file operations and is devoid of the duplicative file operations;
analyzing the time series data to determine whether a subset of the unique file operations includes file-read instructions to copy files corresponding to the subset of unique file operations;
comparing a pattern of the file-read instructions in the time interval to a normal pattern of file-read instructions;
determining, based at least in part on the comparing, that the file-read instructions in the subset of unique file operations are abnormal based at least in part on a deviation between the pattern of the file-read instructions in the time interval and the normal pattern of file-read instructions;
responsive to determining that the file-read instructions in the subset of unique file operations are abnormal, determining that the file system is subject to a threat; and
generating an alert based at least in part on determining that the file system is subject to the threat.
16. The medium of claim 15 , wherein the audit events include information comprising, for each audit event, a user id, a file name, a type of access, and a timestamp.
17. The medium of claim 15 , wherein de-duplicating the audit events is based at least in part on an identification of successive file operations that do not lead to a change in a file state.
18. The medium of claim 15 , wherein the operations further comprise:
generating a finite state machine including one or more file states, the file states including a file open state, a file read state, a file write state, a file read/write state, and a file close state; and
storing the file states in the finite state machine in a key-value object store.
19. The medium of claim 15 , wherein determining that the file-read instructions in the subset of unique file operations are abnormal comprises applying a set of machine learning models to the audit events, the set of machine learning models trained to determine the pattern or a number of the file-read instructions and to compare the pattern or the number of the file-read instructions to the normal pattern of file-read instructions or a normal number of file-read instructions based on features representing a normal or expected behavior of the file system.
20. The medium of claim 18 , wherein de-duplicating the audit events includes maintaining a file system state based on the finite state machine.