Systems and method for rectifying server failure of distributed file systems utilizing predictive logical markers
This disclosure presents a system for rectifying server failures in distributed file systems using predictive logical markers. The system begins by receiving user details, a file name, and a block address via a distributed file system (DFS) client. If previous read information exists for the given file and user, a trained machine learning (ML) model predicts logical markers for file fragments. The ML model then transmits remaining block addresses to the DFS client. Concurrently, the system facilitates a data input stream, communicating predicted block addresses between the DFS client and the ML model. Block location information is retrieved from a distributed ledger via a primary node. The data input stream is processed via a secondary node, allowing efficient rectification of server failures.
1. A system for rectifying server failure of distributed file systems utilizing predictive logical markers, the system comprising:
a processing device;
a non-transitory storage device containing instructions when executed by the processing device, causes the processing device to perform the steps of:
receiving, via a distributed file system (DFS) client, user details, a file name, and a block address;
determining, via the DFS client, that previous read information is available for the file name and the user details;
predicting, via a trained machine learning (ML) model, logical markers for file fragments of a file based on previous read information, the user details, the file name, and the block address;
transmitting, via the ML model, one or more remaining block addresses to the DFS client upon determining the previous read information is available;
facilitating a data input stream, wherein the data input stream comprises a communication of a predicted block address, wherein the data input stream is operatively connected to both the DFS client and the ML model;
retrieving, via a primary node, block location information from a distributed ledger; and
processing the data input stream via a secondary node.
2. The system of claim 1 , wherein the system is further configured to:
determine a failover event in an absence of previous read information; and
transmit all block addresses for the file to the DFS client, allowing for processing of the file.
3. The system of claim 1 , wherein the ML model is trained to predict the logical markers of file fragments dispersed across multiple nodes, using factors including processing speed of each node, file processing methods employed, and characteristics of the file.
4. The system of claim 1 , wherein the ML model is configured to adopt online learning comprising continuous learning and updating of parameters based on new data in real time or near-real time.
5. The system of claim 1 , wherein the user details comprise user identity and user file processing history.
6. The system of claim 1 , wherein the ML model comprises data inputs including a history of file processing events and specific node information including processing speed, file size, and file type.
7. The system of claim 1 , wherein the block addresses sent to the DFS client indicate where, in the file, processing should resume in the event of a failover.
8. A computer program product for rectifying server failure of distributed file systems utilizing predictive logical markers, the computer program product comprising a non-transitory computer-readable medium comprising code causing an apparatus to:
receive, via a distributed file system (DFS) client, user details, a file name, and a block address;
determine, via the DFS client, that previous read information is available for the file name and the user details;
predict, via a trained machine learning (ML) model, logical markers for file fragments of a file based on previous read information, the user details, the file name, and the block address;
transmit, via the ML model, one or more remaining block addresses to the DFS client upon determining the previous read information is available;
facilitate a data input stream, wherein the data input stream comprises a communication of a predicted block address, wherein the data input stream is operatively connected to both the DFS client and the ML model;
retrieve, via a primary node, block location information from a distributed ledger; and
process the data input stream via a secondary node.
9. The computer program product of claim 8 , wherein the code further causes the apparatus to:
determine a failover event in an absence of previous read information; and
transmit all block addresses for the file to the DFS client, allowing for processing of the file.
10. The computer program product of claim 8 , wherein the ML model is trained to predict the logical markers of file fragments dispersed across multiple nodes, using factors including processing speed of each node, file processing methods employed, and characteristics of the file.
11. The computer program product of claim 8 , wherein the ML model is configured to adopt online learning comprising continuous learning and updating of parameters based on new data in real time or near-real time.
12. The computer program product of claim 8 , wherein the user details comprise user identity and user file processing history.
13. The computer program product of claim 8 , wherein the ML model comprises data inputs including a history of file processing events and specific node information including processing speed, file size, and file type.
14. The computer program product of claim 8 , wherein the block addresses sent to the DFS client indicate where, in the file, processing should resume in the event of a failover.
15. A method for rectifying server failure of distributed file systems utilizing predictive logical markers, the method comprising:
receiving, via a distributed file system (DFS) client, user details, a file name, and a block address;
determining, via the DFS client, that previous read information is available for the file name and the user details;
predicting, via a trained machine learning (ML) model, logical markers for file fragments of a file based on previous read information, the user details, the file name, and the block address;
transmitting, via the ML model, one or more remaining block addresses to the DFS client upon determining the previous read information is available;
facilitating a data input stream, wherein the data input stream comprises a communication of a predicted block address, wherein the data input stream is operatively connected to both the DFS client and the ML model;
retrieving, via a primary node, block location information from a distributed ledger; and
processing the data input stream via a secondary node.
16. The method of claim 15 , wherein the method further comprises:
determining a failover event in an absence of previous read information; and
transmitting all block addresses for the file to the DFS client, allowing for processing of the file.
17. The method of claim 15 , wherein the ML model is trained to predict the logical markers of file fragments dispersed across multiple nodes, using factors including processing speed of each node, file processing methods employed, and characteristics of the file.
18. The method of claim 15 , wherein the ML model is configured to adopt online learning comprising continuous learning and updating of parameters based on new data in real time or near-real time.
19. The method of claim 15 , wherein the user details comprise user identity and user file processing history.
20. The method of claim 15 , wherein the ML model comprises data inputs including a history of file processing events and specific node information including processing speed, file size, and file type.