System and method for efficient storage of small files on file-system-based storage devices
Methods and systems for storing and managing large numbers of small files. A data processing system includes clients that generate large numbers be stored on a storage device managed by a File System (FS). An Archive Server (AS) receives multiple files from the client, archives the files in larger archives, and sends the archives to the FS for storage. When requested to read a file, the AS retrieves the archive in which the file is stored, extracts the file and sends it to the requesting client. In other words, the AS communicates with the clients in individual file units, and with the storage device in archive units. The AS is typically constructed as an add-on layer on top of a conventional FS, which enables the FS to handle small files efficiently without modification.
1. A data processing system for facilitating storage of files generated by a client device, comprising: a storage device that is managed by a File System (FS) that allocates storage space on the storage device based on a native allocation unit of storage; and
an archive server in electronic communication with the client device for receiving files from the client device for storage on the storage device, wherein a size of a file received from the client device is smaller than the native avocation unit of storage, wherein the archive server comprises-an archiving processor that is configured to:
receive files from the client device for storage;
group the files from the client device into an archive data structure, wherein the archive server is in electronic communication with more than one client device and is configured to group the flies into more than one archive data structure according to the client device from which the files originated;
receive a request from a particular client device to read a particular file, the request comprising the particular file's identifier;
query the database on the archive server with the identifier to determine a particular archive data structure in which the particular the is stored;
retrieve the particular archive data structure from the storage device;
extract the articular file from the particular archive data structure; and
transmit the particular file to the particular client device;
provide the archive data structure to the storage device for storage after the archive data structure has achieved a predetermined order of magnitude larger than the native allocation unit;
update a database stored on the archive server with information linking an identifier of the file to the archive data structure; and
transmitting the identifier of the file to the client device for future retrieval.
2. The data processing system according to claim 1 , wherein the archive processor is further configured to:
detect that a file received from a client device for storage is a duplicate of a file already stored in an archive data structure;
avoid storing a copy of the duplicate in the storage system by updating the database stored on the archive server with information linking the file already stored in an archive data structure to the file received from the client device; and
transmit the identifier of the file already stored in an archive data structure to the client device for future retrieval.
3. The data processing system according to claim 1 , wherein the archive server is configured to provide a single copy of a file in the archive data structure.
4. A method of storing files generated by a client device in a storage device that is managed by a File System (FS) that allocates storage space on the storage device based on a native avocation unit of storage, comprising:
receiving files from the client device at an archive server for storage on the storage device, wherein a size of a file received from the client device is smaller than the native avocation unit of storage;
grouping the files from the client device into an archive data structure, wherein grouping the files into more than one archive data structure according to the client device from which the files originated;
receive a request from a particular client device to read a particular file, the request comprising the particular file's identifier;
query the database on the archive server with the identifier to determine a particular archive data structure in which the particular file is stored;
retrieve the particular archive data structure from the storage device;
extract the particular file from the particular archive data structure; and
transmit the particular file to the particular client device;
providing the archive data structure to the storage device for storage after the archive data structure has achieved a predetermined order of magnitude larger than the native allocation unit;
updating a database stored on the archive server with information linking an identifier of the file to the archive data structure; and
transmitting the identifier of the file to the client device for future retrieval.
5. The method according to claim 4 , further comprising:
detecting that a file received from a client device for storage is a duplicate of a file already stored in an archive data structure;
avoiding storing a copy of the duplicate in the storage system by updating the database stored on the archive server with information linking the file already stored in an archive data structure to the file received from the client device; and
transmitting the identifier of the file already stored in an archive data structure to the client device for future retrieval.
6. The method according to claim 4 , further comprising providing a single copy of a file in the archive data structure.