System and method for optimizing memory utilization in a distributed file system

1 . A method for optimizing memory utilization in a distributed file system, the method comprising:

receiving a write request to store a data chunk in contiguous storage space in a node of a plurality of nodes in the distributed file system;

monitoring memory usage of a data set across the plurality of nodes in the distributed file system, wherein the data set is partitioned into a plurality of data blocks across the plurality of nodes;

determining a current state of data block replications across the plurality of nodes;

selecting a source node, from the plurality of nodes, to store the data chunk;

determining, based on the monitored memory usage and the current state of data block replications, one or more data blocks in the selected source node to be replaced;

determining, based on the monitored memory usage and the current state of data block replications, one or more target nodes, from the plurality of nodes, to receive the replaced data blocks;

executing a block replacement operation that relocates the one or more data blocks from the selected source node to the one or more target nodes to consolidate contiguous free space in the selected source node sufficient to store the data chunk; and

dynamically adjusting at least one of (a) the selecting of the source node or (b) the determining of the one or more target nodes based on historical write request patterns according to a reinforcement learning model.

2 . The method of claim 1 , wherein the block replacement operation comprises: migrating the one or more data blocks from selected source node with memory usage exceeding a predefined threshold to the one or more target nodes with available memory below the predefined threshold.

3 . The method of claim 2 , wherein the block replacement operation comprises: rebalancing a distribution of the one or more data blocks to provide optimal memory utilization across the plurality of nodes.

4 . The method of claim 1 , wherein dynamically adjusting the selecting of the source node comprises selecting a node with minimal memory space that is sufficient to accommodate a data block and minimize wasted memory space.

5 . The method of claim 1 , wherein dynamically adjusting the selecting of the source node comprises selecting a node with maximum memory space to provide a larger remainder of unused memory space for future use.

6 . The method of claim 1 , wherein dynamically adjusting the selecting of the source node comprises selecting a first node that has sufficient memory space to accommodate a data block and minimizing time taken to find a suitable node to store the data chunk.

7 . The method of claim 1 , comprising determining one or more target nodes, from the plurality of nodes, to receive the replaced data blocks based on a best fit algorithm and selecting a source node with minimal space sufficient to accommodate a data block to minimize wasted space.

8 . The method of claim 1 , comprising determining one or more target nodes, from the plurality of nodes, to receive the replaced data blocks based on a worst fit algorithm and selecting a source node with maximum space to leave a larger remainder for future use.

9 . The method of claim 1 , comprising determining one or more target nodes, from the plurality of nodes, to receive the replaced data blocks based on a first fit algorithm and selecting a first node that has sufficient space to accommodate a data block to minimize time taken to find a suitable node.

10 . A system for optimizing memory utilization in a distributed file system comprising a processor and a non-transitory computer-readable medium coupled to the processor, the non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to:

receive a write request to store a data chunk in contiguous storage space in a node of a plurality of nodes in the distributed file system;

monitor memory usage of a data set across the plurality of nodes in the distributed file system, wherein the data set is partitioned into a plurality of data blocks across the plurality of nodes;

determine a current state of data block replications across the plurality of nodes;

select a source node, from the plurality of nodes, to store the data chunk;

determine, based on the monitored memory usage and the current state of data block replications, one or more data blocks in the selected source node to be replaced;

determine, based on the monitored memory usage and the current state of data block replications, one or more target nodes, from the plurality of nodes, to receive the replaced data blocks;

execute a block replacement operation that relocates the one or more data blocks from the selected source node to the one or more target nodes to consolidate contiguous free space in the selected source node sufficient to store the data chunk; and

dynamically adjust selection of at least one of (a) the selecting of the source node or (b) the determining of the one or more target nodes based on historical write request patterns according to a reinforcement learning model.

11 . The system of claim 10 , wherein the non-transitory computer-readable medium stores additional instructions that, when executed by the processor, cause the processor to migrate the one or more data blocks from selected source node with memory usage exceeding a predefined threshold to the one or more target nodes with available memory below the predefined threshold as part of the block replacement operation.

12 . The system of claim 10 , wherein the non-transitory computer-readable medium stores additional instructions that, when executed by the processor, cause the processor to rebalance a distribution of the one or more data blocks to provide optimal memory utilization across the plurality of nodes as part of the block replacement operation.

13 . The system of claim 10 , wherein the non-transitory computer-readable medium stores additional instructions that, when executed by the processor, cause the processor to:

determine one or more target nodes, from the plurality of nodes, to receive the replaced data blocks based on a best fit algorithm and selecting a source node with minimal space sufficient to accommodate a data block to minimize wasted space;

determine one or more target nodes, from the plurality of nodes, to receive the replaced data blocks based on a worst fit algorithm and selecting a source node with maximum space to leave a larger remainder for future use; or

determine one or more target nodes, from the plurality of nodes, to receive the replaced data blocks based on a first fit algorithm and selecting a first node that has sufficient space to accommodate a data block to minimize time taken to find a suitable node.