Predictive real-time anti-virus scanning
Implementations are provided herein for enhanced antivirus scanning in distributed file storage systems. In embodiments, we disclose using predictive, heuristic data to determine prospectively a sequence of files that may be called by a client based upon relationships between an initial filed called by a client and other files stored in the distributed file system. The sequence of files likely to be called in the future could, in some embodiments, be determined using machine language trained to analyze past client behavior patterns. Once a predictive file sequence has been determined, embodiments herein send files likely to be called in the very near future to an antivirus server for processing during a time period that overlaps the AV processing period for the original file that was called.
1. A method comprising:
receiving a first access request at a node of a distributed file system from a client for a first file stored on the node;
in response to the first access request being received, determining whether the first file must undergo an anti-virus scan prior to allowing access to the first file based on determining when the first file was last scanned for virus and whether the virus has been updated since the first file was last scanned;
in response to determining that the first file does not need to undergo the anti-virus scan, providing the client access to the first file;
in response to determining that the first file must undergo the anti-virus scan prior to allowing access to the first file, placing the first file in a real-time anti-virus scan queue;
in response to the placement of the first file in the real-time anti-virus scan queue, predicting at least a second file and a third file, which will be requested after the first file, wherein the prediction is based on a heuristics data set, the heuristics data set is compiled using machine learning to observe interrelationships between files stored in the distributed file system and evaluate a sequence of repetitive file calls made by the client, the heuristics data set including a statistically significant likelihood that a second access request will be received from the client for the at least second file and third file based on a past pattern showing the at least second file and third file being requested after the first file, wherein the second access request will be received after the first access request;
in response to the prediction, placing the at least second file and third file in a predictive anti-virus scan queue; and
in response to the placement of the first file in the real-time anti-virus scan queue and the placement of the at least second file and third file in the predictive anti-virus scan queue, performing anti-virus scanning on the first file and the at least second file and third file, wherein the anti-virus scanning on the first file and the at least second file and third file overlap at a point in time.
2. The method of claim 1 , further comprising consulting an anti-virus table stored in a memory in order to determine if the first file must undergo the anti-virus scan.
3. The method of claim 1 , wherein the heuristics data set comprises a plurality of access patterns associated with the first file.
4. The method of claim 1 , wherein the prediction is based on an interrelationship between the first file and the at least second file and third file.
5. A system comprising at least one storage device and at least one hardware processor, the hardware processor configured to:
receive a first access request at a node of a distributed file system from a client for a first file stored on the node;
in response to the first access request being received, determine whether the first file must undergo an anti-virus scan prior to allowing access to the first file based on determining when the first file was last scanned for virus and whether the virus has been updated since the first file was last scanned;
in response to determining that the first file does not need to undergo the anti-virus scan, providing the client access to the first file;
in response to determining that the first file must undergo the anti-virus scan prior to allowing access to the first file, place the first file in a real-time anti-virus scan queue;
in response to the placement of the first file in the real-time anti-virus scan queue, predict at least a second file and a third file, which will be requested after the first file, wherein the prediction is based on a heuristics data set, the heuristics data set is compiled using machine learning to observe interrelationships between files stored in the distributed file system and evaluate a sequence of repetitive file calls made by the client, the heuristics data set including a statistically significant likelihood that a second access request will be received from the client for the at least second file and third file based on a past pattern showing the at least second file and third file being requested after the first file wherein the second access request will be received after the first access request;
in response to the prediction, place the at least second file and third file in a predictive anti-virus scan queue; and
in response to the placement of the first file in the real-time anti-virus scan queue and the placement of the at least second file and third file in the predictive anti-virus scan queue, perform anti-virus scanning on the first file and the at least second file and third file, wherein the anti-virus scanning on the first file and the at least second file and third file overlap at a point in time.
6. The system of claim 5 , the hardware processor further configured to consult an anti-virus table stored on the node in order to determine if the first file must undergo the anti-virus scan.
7. The system of claim 5 , wherein the heuristics data set comprises a plurality of access patterns associated with the first file.
8. The system of claim 5 , wherein the prediction is based on an interrelationship between the first file and the at least second file and third file.
9. A non-transitory computer readable medium with program instructions stored thereon, the program instructions when executed by a processor cause the processor to perform operations comprising:
receive a first access request at a node of a distributed file system from a client for a first file stored on the node;
in response to the first access request being received, determine whether the first file must undergo an anti-virus scan prior to allowing access to the first file based on determining when the first file was last scanned for virus and whether the virus has been updated since the first file was last scanned;
in response to determining that the first file does not need to undergo the anti-virus scan, providing the client access to the first file;
in response to determining that the first file must undergo the anti-virus scan prior to allowing access to the first file, place the first file in a real-time anti-virus scan queue;
in response to the placement of the first file in the real-time anti-virus scan queue, predict at least a second file and a third file, which will be requested after the first file, wherein the prediction is based on a heuristics data set, the heuristics data set is compiled using machine learning to observe interrelationships between files stored in the distributed file system and evaluate a sequence of repetitive file calls made by the client, the heuristics data set including a statistically significant likelihood that a second access request will be received from the client for the at least second file and third file based on a past pattern showing the at least second file and third file being requested after the first file, wherein the second access request will be received after the first access request;
in response to the prediction, place the at least second file and third file in a predictive anti-virus scan queue; and
in response to the placement of the first file in the real-time anti-virus scan queue and the placement of the at least second file and third file in the predictive anti-virus scan queue, perform anti-virus scanning on the first file and the at least second file and third file, wherein the anti-virus scanning on the first file and the at least second file and third file overlap at a point in time.
10. The non-transitory computer readable medium of claim 9 , wherein the operations further comprise consulting an antivirus table stored in a memory in order to determine if the first file must undergo the anti-virus scan.
11. The non-transitory computer readable medium of claim 9 , wherein the heuristics data set comprises a plurality of access patterns associated with the first file.
12. The non-transitory computer readable medium of claim 9 , wherein the prediction is based on an interrelationship between the first file and the at least second file and third file.