Load based file allocation among a plurality of storage devices

1. A method for balancing loads among a plurality of data storage devices, comprising:

receiving a request from a user device to download a data file;

identifying a first set of storage devices containing the requested data file, wherein the first set comprises one or more storage devices;

identifying a first storage device from the set that is currently least busy among the storage devices in the first set, and directing the first storage device to serve the request;

determining if the first storage device is too busy;

if the first storage device is too busy, copying the requested data file to a second storage device which did not previously store the data file, thereby adding the second storage device to the first set of storage devices containing the requested data file; and

determining whether one or more redundant copies of the requested file from the first set of storage devices should be deleted based at least in part on a number of current access requests for the file, wherein determining whether one or more redundant copies of the requested file should be deleted comprises determining whether a quotient, provided by a total number of current access requests for the file divided by a total number of copies of the files stored in the first set, is lower than a predetermined threshold, wherein if the quotient is lower than the predetermined threshold, the one or more redundant copies are designated to be deleted.

2. The method of claim 1 wherein determining if the first storage device is too busy comprises determining whether a number of current access requests being served by the first storage device exceeds a predetermined threshold value.

3. The method of claim 1 further comprising determining how many redundant copies to delete based on the formula: N=CFC−(T/CAC), where N is the number of redundant copies to delete, CFC is the total number of copies of the requested file stored in the set, T is a predetermined threshold selected based on a desired average load upon the set of storage devices, and CAC is the total number of current access requests for the data file.

4. The method of claim 3 further comprising:

designating the N redundant files to be deleted as “offline” so they cannot be accessed; and

deleting the N redundant files as a backend process which is transparent to end users who access files from the first set of storage devices.

5. The method of claim 3 wherein the N redundant copies designated for deletion comprise the N most recently stored copies of the data file.

6. A system for balancing loads among a plurality of storage servers within a storage node, comprising:

a node manager server for receiving a request from a user device to download a data file;

a plurality of storage servers contained within a first storage node for storing one or more copies of the data file;

a transfer server coupled to the plurality of storage servers for downloading the data file requested by users;

wherein the node manager server comprises:

a first module for receiving the request from the user device to download the data file;

a second module for identifying a first set of storage servers among the plurality of storage servers in the node containing the requested data file, wherein the first set comprises one or more storage servers;

a third module for identifying a first storage server from the first set that is currently least busy among the storage servers in the first set, and directing the first storage server to serve the request;

a fourth module for determining if the first storage device is too busy, wherein if the first storage device is too busy, copying the requested data file to a second storage server which did not previously store the data file, thereby adding the second storage server to the first set of storage servers containing the requested data file; and

a fifth module for determining whether one or more redundant copies of the requested file from the first set of storage servers should be deleted based at least in part on a number of current access requests for the file, wherein the fifth module comprises a sixth module for determining whether a quotient, provided by a total number of current access requests for the file divided by a total number of copies of the files stored in the first set, is lower than a predetermined threshold, wherein if the quotient is lower than the predetermined threshold, the one or more redundant copies are designated to be deleted.

7. The system of claim 6 wherein the fifth module further determines whether a number of current access requests being served by the first storage server exceeds a predetermined threshold value.

8. The system of claim 6 further comprising a seventh module for determining how many redundant copies to delete based on the formula: N=CFC−(T/CAC), where N is the number of redundant copies to delete, CFC is the total number of copies of the requested file stored in the set, T is a predetermined threshold selected based on a desired average load upon the set of storage devices, and CAC is the total number of current access requests for the data file.

9. The system of claim 8 further comprising:

an eighth module for designating the N redundant files to be deleted as “offline” so they cannot be accessed; and

a ninth module for deleting the N redundant files as a backend process which is transparent to the user.

10. The system of claim 8 wherein the N redundant copies designated for deletion comprise the N most recently stored copies of the data file.

11. A non-transient computer readable medium storing computer executable instructions that when executed perform a process for balancing loads among a plurality of storage servers within a storage node, the instructions comprising:

a first code module for receiving a request from a user device to download a data file;

a second code module for identifying a first set of storage servers among a plurality of storage servers in a first node containing the requested data file, wherein the first set comprises one or more storage servers;

a third code module for identifying a first storage server from the first set that is currently least busy among the storage servers in the first set, and directing the first storage server to serve the request;

a fourth code module for determining if the first storage device is too busy, wherein if the first storage device is too busy, copying the requested data file to a second storage server which did not previously store the data file, thereby adding the second storage server to the first set of storage servers containing the requested data file, wherein the fourth code module comprises a fifth code module for determining whether a number of current access requests being served by the first storage server exceeds a predetermined threshold value; and

wherein the fifth code module further determines whether one or more redundant copies of the requested file from the first set of storage servers should be deleted based at least in part on a number of current access requests for the file, wherein the fifth code module comprises a sixth code module for determining whether a quotient, provided by a total number of current access requests for the file divided by a total number of copies of the files stored in the first set, is lower than a predetermined threshold, wherein if the quotient is lower than the predetermined threshold, the one or more redundant copies are designated to be deleted.

12. The computer readable medium of claim 11 further comprising a seventh code module for determining how many redundant copies to delete based on the formula: N=CFC−(T/CAC), where N is the number of redundant copies to delete, CFC is the total number of copies of the requested file stored in the set, T is a predetermined threshold selected based on a desired average load upon the set of storage devices, and CAC is the total number of current access requests for the data file.

13. The computer readable medium of claim 12 further comprising:

an eighth code module for designating the N redundant files to be deleted as “offline” so they cannot be accessed; and

a ninth code module for deleting the N redundant files as a backend process which is transparent to the user.

14. The computer readable medium of claim 12 wherein the N redundant copies designated for deletion comprise the N most recently stored copies of the data file.