Machine learning architecture for detecting malicious files using stream of data
The present application discloses a method, system, and computer system for classifying stream data at an edge device. The method includes obtaining a stream of a file at the edge device, aligning a predetermined amount of data in chunks associated with the stream of the file, processing a plurality of aligned chunks associated with the stream of the file using a machine learning model, and classifying, at the edge device, the file based at least in part on a classification of the plurality of aligned chunks.
1. A system for performing classification at an edge device, comprising:
one or more processors configured to:
obtain a stream of a file at the edge device;
align a predetermined amount of data in chunks associated with the stream of the file, wherein aligning the predetermined amount of data in chunks associated with the stream of the file comprises:
determining an nth file segment based at least in part on associating a predetermined amount of an ith chunk with a predetermined amount of a jth chunk, wherein i and j are positive integers, and j is greater than i;
process a plurality of aligned chunks associated with the stream of the file using a machine learning model; and
classify, at the edge device, the file based at least in part on a classification of the plurality of aligned chunks, wherein the file is classified based at least in part on a classification of the nth file segment; and
a memory coupled to the one or more processors and configured to provide the one or more processors with instructions.
2. The system of claim 1 , wherein the edge device is a network device.
3. The system of claim 1 , wherein the edge device is an inline security entity.
4. The system of claim 1 , wherein the machine learning model is configured to classify whether the file is malicious.
5. The system of claim 1 , wherein the file is classified using the machine learning model before an entirety of the file is processed.
6. The system of claim 1 , wherein a first chunk comprises overhead associated with the file.
7. The system of claim 1 , wherein:
determining an nth file segment based at least in part on associating a predetermined amount of an ith chunk with a predetermined amount of a jth chunk comprises:
determining a first file segment based at least in part on associating a predetermined amount of a first chunk with a predetermined amount of a second chunk;
processing the plurality of aligned chunks comprises querying the machine learning model based on the first file segment; and
the file is classified based at least in part on a classification of the first file segment.
8. The system of claim 1 , wherein:
processing the plurality of aligned chunks comprises querying the machine learning model based on the nth file segment.
9. The system of claim 8 , wherein the nth file segment comprises a predetermined number of bytes.
10. The system of claim 8 , wherein the nth file segment comprises 1500 bytes.
11. The system of claim 8 , wherein the one or more processors are configured to select the predetermined number of bytes from among a set of preset numbers of bytes, the predetermined number of bytes being selected based on a packet size of the file.
12. The system of claim 1 , wherein the aligning the predetermined amount of data in chunks adjusts for file overhead in a first chunk and ensures that classification comprises processing a same number of bytes of the file for each alignment-adjusted chunk.
13. The system of claim 1 , wherein using the machine learning model to classify the file based on a predefined number of bytes is deterministic.
14. The system of claim 1 , wherein the file is determined to be malicious if a prediction obtained from the machine learning model exceeds a predefined malicious threshold.
15. The system of claim 14 , wherein the file is determined to be malicious after an nth file segment is processed using the machine learning model, n corresponding to a positive integer that is less than a total number of chunks in the file.
16. The system of claim 14 , wherein the predefined malicious threshold is constant for each file segment in the file.
17. The system of claim 14 , wherein the predefined malicious threshold is dynamic across classification of file segment in the file.
18. The system of claim 14 , wherein in response to determining that the file is malicious, an active measure for malicious files is implemented.
19. The system of claim 1 , wherein the machine learning model is trained using a deep learning process.
20. The system of claim 15 , wherein the deep learning process comprises a convolutional neural network.
21. The system of claim 20 , wherein the convolutional neural network uses a kernel size less than 12.
22. The system of claim 20 , wherein the convolutional neural network uses a kernel size of 8.
23. The system of claim 1 , wherein a number of characters for which state information is stored for evaluation of aligned chunks is based at least in part on a kernel size of the convolutional neural network.
24. The system of claim 1 , wherein the machine learning model is an XGBoost model.
25. A method for performing classification at an edge device, comprising:
obtaining, by one or more processors, a stream of a file at the edge device;
aligning a predetermined amount of data in chunks associated with the stream of the file, wherein aligning the predetermined amount of data in chunks associated with the stream of the file comprises:
determining an nth file segment based at least in part on associating a predetermined amount of an ith chunk with a predetermined amount of a jth chunk, wherein i and i are positive integers, and j is greater than i;
processing a plurality of aligned chunks associated with the stream of the file using a machine learning model; and
classifying, at the edge device, the file based at least in part on a classification of the plurality of aligned chunks, wherein the file is classified based at least in part on a classification of the nth file segment.
26. A computer program product embodied in a non-transitory computer readable medium for performing classification at an edge device, and the computer program product comprising computer instructions for:
obtaining, by one or more processors, a stream of a file at the edge device;
aligning a predetermined amount of data in chunks associated with the stream of the file, wherein aligning the predetermined amount of data in chunks associated with the stream of the file comprises:
determining an nth file segment based at least in part on associating a predetermined amount of an ith chunk with a predetermined amount of a jth chunk, wherein i and j are positive integers, and j is greater than i;
processing a plurality of aligned chunks associated with the stream of the file using a machine learning model; and
classifying, at the edge device, the file based at least in part on a classification of the plurality of aligned chunks, wherein the file is classified based at least in part on a classification of the nth file segment.
27. The system of claim 1 , wherein the predetermined amount of data is aligned in chunks associated with the stream of the file based at least in part on a predefined chunk size and a file segment size for segments of the stream of the file.