Machine learning architecture for detecting malicious files using stream of data
The present application discloses a method, system, and computer system for classifying stream data at an edge device. The method includes obtaining a stream of a file at the edge device, processing a set of chunks associated with the stream of the file using a machine learning model, and classifying, at the edge device, the file before processing an entirety of the file.
1. A system for performing classification at an edge device, comprising:
one or more processors configured to:
obtain a stream of a file at the edge device;
process a set of chunks associated with the stream of the file using a machine learning model; and
classify, at the edge device, the file before processing an entirety of the file, wherein:
the file is determined to be malicious if a prediction obtained from the machine learning model exceeds a predefined malicious threshold;
the predefined malicious threshold is different across classification of chunks in the file; and
the predefined malicious threshold is lower for a first chunk than for a jth chunk, and j is a positive integer greater than 1; and
a memory coupled to the one or more processors and configured to provide the one or more processors with instructions.
2. The system of claim 1 , wherein the edge device is a network device.
3. The system of claim 1 , wherein the edge device is an inline security entity.
4. The system of claim 1 , wherein the machine learning model is configured to classify whether the file is malicious.
5. The system of claim 1 , wherein the machine learning model is configured to classify whether the file is copyright or protected material.
6. The system of claim 1 , wherein the machine learning model is configured to classify whether the file is health or financial data.
7. The system of claim 1 , wherein the file is determined to be malicious after an nth chunk is processed using the machine learning model, n corresponding to a positive integer that is less than a total number of chunks in the file.
8. The system of claim 1 , wherein the predefined malicious threshold is constant for a plurality of chunks in the file.
9. The system of claim 1 , wherein in response to determining that the file is malicious, an active measure for malicious files is implemented.
10. The system of claim 9 , wherein the active measure includes dropping or blocking remaining chunks associated with the file.
11. The system of claim 1 , wherein each chunk corresponds to m bytes, and m is a positive integer.
12. The system of claim 1 , wherein the machine learning model is trained using a deep learning process.
13. The system of claim 12 , wherein the deep learning process comprises a convolutional neural network.
14. The system of claim 12 , wherein the machine learning model is trained based at least in part on a recursive neural network, and a max pooling operation is performed to maintain state information across at least a subset of chunks associated with the file.
15. The system of claim 1 , wherein the model is trained with respect to an entire file.
16. A method for performing classification at an edge device, comprising:
obtaining, by one or more processors, a stream of a file at the edge device;
processing a set of chunks associated with the stream of the file using a machine learning model; and
classifying, at the edge device, the file before processing an entirety of the file;
wherein:
the file is determined to be malicious if a prediction obtained from the machine learning model exceeds a predefined malicious threshold;
the predefined malicious threshold is different across classification of chunks in the file; and
the predefined malicious threshold is lower for a first chunk than for a jth chunk, and j is a positive integer greater than 1.
17. A computer program product embodied in a non-transitory computer readable medium for performing classification at an edge device, and the computer program product comprising computer instructions for:
obtaining, by one or more processors, a stream of a file at the edge device;
processing a set of chunks associated with the stream of the file using a machine learning model; and
classifying, at the edge device, the file before processing an entirety of the file;
wherein:
the file is determined to be malicious if a prediction obtained from the machine learning model exceeds a predefined malicious threshold;
the predefined malicious threshold is different across classification of chunks in the file; and
the predefined malicious threshold is lower for a first chunk than for a jth chunk, and j is a positive integer greater than 1.
18. A system for performing classification at an edge device, comprising:
one or more processors configured to:
obtain a stream of a file at the edge device;
process a set of chunks associated with the stream of the file using a machine learning model; and
classify, at the edge device, the file before processing an entirety of the file; and
a memory coupled to the one or more processors and configured to provide the one or more processors with instructions;
wherein:
the machine learning model is trained using a deep learning process; and
the machine learning model is trained based at least in part on a recursive neural network, and a max pooling operation is performed to maintain state information across at least a subset of chunks associated with the file.
19. A method for performing classification at an edge device, comprising:
obtaining, by one or more processors, a stream of a file at the edge device;
processing a set of chunks associated with the stream of the file using a machine learning model; and
classifying, at the edge device, the file before processing an entirety of the file;
wherein:
the machine learning model is trained using a deep learning process; and
the machine learning model is trained based at least in part on a recursive neural network, and a max pooling operation is performed to maintain state information across at least a subset of chunks associated with the file.