Directory aggregation for files distributed over a plurality of servers in a switched file system
View Patent ↗A file switch, logically positioned between client computers and file servers in a computer network, distributes user files among multiple file servers using an aggregated directory mechanism. A hierarchical directory structure is created on the file servers and used to store metadata files, which store metadata for each user file to indicate where data files, containing portions of the user file, are stored. The file switch automatically spreads the data files and metadata files over a large number of distinct directories on multiple file servers, preventing large number of data files from being stored in a single directory on a single file server. In response to a directory enumeration request from a client computer, one or more directories of metadata files on one or more of the file servers are enumerated, instead of enumerating the data file that store the user file portions.
1. A file switch for use in a computer network having a group of file servers and a plurality of client computers, wherein the plurality of client computers are coupled to the group of file servers through the file switch, the file switch comprising:
at least one processing unit for executing computer programs;
at least one port for exchanging information with the file servers and client computers, the information exchanged including information concerning a specified user file;
a file aggregation module including one or more computer programs, the computer programs including instructions for:
receiving, at the file switch, a user request for a directory listing of user files for a specified directory;
accessing a directory of metafiles corresponding to all user files in the specified directory, each metafile from the directory of metafiles storing information identifying a set of file servers that store respective defined one or more stripe portions of the specified user file, each of the one or more stripe portions comprising one or more strips that have a respective size, wherein the directory listing of metafiles is obtained from a particular file server of the group of file servers in which the metafiles are stored, the particular file server selected at least based upon the size of the one or more strips;
constructing, at the file switch, a directory listing of the user files from the directory of metafiles; and
sending the directory listing from the file switch to a particular client computer of the plurality of client computers.
2. The file switch of claim 1 , wherein the directory listing of metafiles includes, for each metafile, an aggregated file size for the user file corresponding to the metafile.
3. The file switch of claim 2 , wherein the aggregated file size is stored in a data field of the directory listing of the metafiles.
4. The file switch of claim 1 , wherein the computer programs of the file aggregation module include instructions for updating the directory of metafiles, whenever an operation is performed on any user file in the specified directory that changes the aggregated file size of the user file, so as to indicate the changed aggregated file size of the user file.
5. The file switch of claim 1 , wherein the specified directory has an associated file path; and
the directory of metafiles corresponding to the user files for the specified directory has a file path corresponding to the file path of the specified directory.
6. A method of managing a directory listing of files in a computer network, comprising:
coupling a plurality of client computers to a group of file servers through a file switch in a computer network;
receiving, at the file switch, a user request for a directory listing of user files for a specified directory;
accessing a directory of metafiles corresponding to all user files in the specified directory, each metafile storing information identifying a set of file servers that store respective defined one or more stripe portions of the specified user file, each of the one or more stripe portions comprising one or more strips that have a respective size, wherein the directory listing of metafiles is obtained from a particular file server of the group of file servers in which the metafiles are stored, the particular file server selected at least based upon the size of the one or more strips;
constructing, at the file switch, a directory listing of the user files from the directory of metafiles; and
sending the directory listing from the file switch to a particular client computer of the plurality of client computers.
7. A file switch for use in a computer network having a group of file servers and a plurality of client computers, wherein the plurality of client computers are coupled to the group of file servers through the file switch, the file switch comprising:
at least one processing unit for executing computer programs;
at least one port for exchanging information with the file servers and client computers, the information exchanged including information concerning a specified user file;
a file aggregation module including one or more computer programs, the computer programs including instructions for:
receiving, at the file switch, a user request for a directory listing of user files for a specified directory;
accessing, in each of a plurality of file servers of the group of file servers, a respective directory of metafiles corresponding to all user files in the specified directory, each metafile storing information identifying a set of file servers that store respective defined one or more stripe portions of the specified user file, each of the one or more stripe portions comprising one or more strips that have a respective size, wherein at least one of the plurality of file servers is selected at least based upon the size of the one or more strips;
aggregating, at the file switch, the accessed directories of metafiles to construct an aggregated directory listing of the user files; and
sending the aggregated directory listing from the file switch to a particular client computer of the plurality of client computers.
8. The file switch of claim 7 , wherein each respective directory listing of metafiles includes, for each metafile listed therein, an aggregated file size for the user file corresponding to the metafile.
9. The file switch of claim 8 , wherein the aggregated file size is stored in a data field of the respective directory listing of the metafiles.
10. The file switch of claim 7 , the computer programs of the file aggregation module including instructions, activated when an operation is performed on a user file in the specified directory that changes the aggregated file size of the user file, for updating the corresponding directory of metafiles to indicate the changed aggregated file size of the user file.
11. The file switch of claim 7 , wherein
the specified directory has an associated file path; and
each respective directory of metafiles corresponding to the user files for the specified directory has a file path corresponding to the file path of the specified directory.
12. A method of managing a directory listing of files in a computer network, comprising:
coupling a plurality of client computers to a group of file servers through a file switch in a computer network;
receiving, at the file switch, a user request for a directory listing of user files for a specified directory;
accessing, in each of a plurality of file servers of the group of file servers, a respective directory of metafiles corresponding to all user files in the specified directory, each metafile storing information identifying a set of file servers that store respective defined one or more stripe portions of the specified user file, each of the one or more stripe portions comprising one or more strips that have a respective size, wherein at least one of the plurality of file servers is selected at least based upon the size of the one or more strips;
aggregating, at the file switch, the accessed directories of metafiles to construct an aggregated directory listing of the user files; and
sending the aggregated directory listing from the file switch to a particular client computer of the plurality of client computers.
13. A file switch for use in a computer network having a group of file servers and a plurality of client computers, wherein the plurality of client computers are coupled to the group of file servers through the file switch, the file switch comprising:
at least one processing unit for executing computer programs;
at least one port for exchanging information with the file servers and client computers, the information exchanged including information concerning a specified user file;
a file aggregation module including one or more computer programs, the computer programs including instructions for:
determining, at the file switch, in accordance with a predefined set of aggregation rules, a set of file servers from the group of file servers for storing the user file;
for each stripe portion of the user file to be stored in a respective one of the determined set of servers, generating, at the file switch, a file path as a function of a file name of the user file, the file path including at least three directory levels below a root path of the file server, wherein each stripe portion comprises one or more strips that have a respective size, at least one of the determined set of servers being selected at least based upon the size of the one or more strips; and
storing each said portion of the user file in the respective one of the determined set of servers at the generated file path.
14. The file switch of claim 13 , wherein the computer programs of the file aggregation module include instructions for generating a globally unique identifier that is unique with respect to globally unique identifiers for all user file portions stored in the group of file servers, extracting a plurality of file path segments from the globally unique identifier, and concatenating the plurality of file path segments to form at least a portion of the file path.
15. The file switch of claim 13 , wherein the computer programs of the file aggregation module include instructions for generating the file path as a function of the file name of the user file, a value corresponding to the file portion for which the file path is being generated, and a value corresponding to the respective server in which the file portion is to be stored.
16. A method of managing a directory listing of files in a computer network, comprising:
coupling a group of client computers to a group of file servers through a file switch in a computer network;
determining, at the file switch, in accordance with a predefined set of aggregation rules, a set of file servers from the group of file servers for storing the user file;
for each stripe portion of the user file to be stored in a respective one of the determined set of servers, generating, at the file switch, a file path as a function of a file name of the user file, the file path including at least three directory levels below a root path of the file server, wherein each stripe portion comprises one or more strips that have a respective size, at least one of the determined set of servers being selected at least based upon the size of the one or more strips; and
storing each said portion of the user file in the respective one of the determined set of servers at the generated file path.