System and method for file archiving using machine learning
Methods for file archiving using machine learning are disclosed herein. An exemplary method comprises archiving a first file of a plurality of files from a storage server to a tiered storage system, training a machine learning module based on file access operations for the plurality of files, determining one or more rules for predicting access to the archived files using the machine learning module, determining a prediction of access of the archived file based on the one or more rules and retrieving the archived file from the tiered storage system into a file cache in the storage server based on the prediction of access.
1. A method for file archiving using machine learning, comprising:
archiving a first file of a plurality of files as an archived file from a storage server to a tiered storage system;
generating a reparse point in the storage server corresponding to the archived first file;
training a machine learning module based on file access operations for the plurality of files;
determining one or more rules for predicting access to the archived first file using the machine learning module, wherein the one or more rules include a causal rule indicating that a likelihood that the first file will be accessed by a user of the storage server increases when the user accesses a second file at the storage server;
determining a prediction of access of the archived first file based on the one or more rules, wherein determining the prediction of access comprises at least detecting user access of the second file at the storage server; and
retrieving the archived first file from the tiered storage system into a file cache in the storage server based on the prediction of access;
receiving, by a file system filter driver, a file request to access the archived first file;
determining a location of the archived file among the tiered storage system and the file cache based on the reparse point; and
retrieving data of the archived first file from the determined location.
2. The method of claim 1 , further comprising:
receiving and servicing a file request to access the archived first file from the file cache, wherein the archived first file was retrieved from the tiered storage system and stored in the file cache prior to receiving the file request.
3. The method of claim 1 , wherein archiving the first file of the plurality of files from the storage server to the tiered storage system comprises:
scanning the plurality of files from the storage server to determine a demand prediction for each of the plurality of files; and
archiving the plurality of files including the first file to the tiered storage system based on the respective demand predictions.
4. The method of claim 3 , further comprising:
training the machine learning module based on archiving the plurality of files; and
archiving the second file of the plurality of files from the storage server to the tiered storage system based on the machine learning module.
5. The method of claim 1 , wherein the one or more rules predict access of the plurality of files based on archival operations and file access operations on the storage server.
6. The method of claim 1 , further comprising:
modifying or removing a rule from the one or more rules based on archival operations and file access operations on the storage server.
7. The method of claim 1 , further comprising:
maintaining a list of file placeholders on the storage server, wherein each file placeholder is a reference to a file in the tiered storage system.
8. The method of claim 7 , wherein the file placeholder is the reparse point.
9. A system for file archiving using machine learning, comprising:
a hardware processor configured to:
archive a first file of a plurality of files as an archived file from a storage server to a tiered storage system;
generate a reparse point in the storage server corresponding to the archived first file;
train a machine learning module based on file access operations for the plurality of files;
determine one or more rules for predicting access to the archived first file using the machine learning module, wherein the one or more rules include a causal rule indicating that a likelihood that the first file will be accessed by a user of the storage server increases when the user accesses a second file at the storage server;
determine a prediction of access of the archived first file based on the one or more rules, wherein determining the prediction of access comprises at least detecting user access of the second file at the storage server; and
retrieve the archived first file from the tiered storage system into a file cache in the storage server based on the prediction of access;
receive, by a file system filter driver, a file request to access the archived first file;
determine a location of the archived file among the tiered storage system and the file cache based on the reparse point; and
retrieve data of the archived first file from the determined location.
10. The system of claim 9 , wherein the processor is further configured to:
receive and service a file request to access the archived first file from the file cache, wherein the archived first file was retrieved from the tiered storage system and stored in the file cache prior to receiving the file request.
11. The system of claim 9 , wherein archiving the first file of the plurality of files from the storage server to the tiered storage system comprises:
scanning the plurality of files from the storage server to determine a demand prediction for each of the plurality of files; and
archiving the plurality of files including the first file to the tiered storage system based on the respective demand predictions.
12. The system of claim 9 , wherein the processor is further configured to:
train the machine learning module based on archiving the plurality of files; and
archive the second file of the plurality of files from the storage server to the tiered storage system based on the machine learning module.
13. The system of claim 9 , wherein the one or more rules predict access of the plurality of files based on archival operations and file access operations on the storage server.
14. The system of claim 9 , wherein the processor is further configured to:
modify or removing a rule from the one or more rules based on archival operations and file access operations on the storage server.
15. The system of claim 9 , wherein the processor is further configured to:
maintain a list of file placeholders on the storage server, wherein each file placeholder is a reference to a file in the tiered storage system.
16. The system of claim 15 , wherein the file placeholder is the reparse point.
17. A computer-readable medium, storing instructions thereon for file archiving using machine learning, the instructions comprising:
archiving a first file of a plurality of files as an archived file from a storage server to a tiered storage system;
generating a reparse point in the storage server corresponding to the archived first file;
training a machine learning module based on file access operations for the plurality of files;
determining one or more rules for predicting access to the archived first file using the machine learning module, wherein the one or more rules include a causal rule indicating that a likelihood that the first file will be accessed by a user of the storage server increases when the user accesses a second file at the storage server;
determining a prediction of access of the archived first file based on the one or more rules, wherein determining the prediction of access comprises at least detecting user access of the second file at the storage server; and
retrieving the archived first file from the tiered storage system into a file cache in the storage server based on the prediction of access;
receiving, by a file system filter driver, a file request to access the archived first file;
determining a location of the archived file among the tiered storage system and the file cache based on the reparse point; and
retrieving data of the archived first file from the determined location.