Application aware deduplication allowing random access to compressed files
A file is received from a client for storage at a deduplication file system. The file is in an archive file format that is used by an application on the client. The file includes subfiles compressed together in the file according to the archive file format, local headers corresponding to the subfiles, and a central directory used by the application to locate information stored in the file. The file is decompressed to store the subfiles separately. A metadata file is created that describes a structure of the file. The metadata file includes the local headers, central directory, pointers to the subfiles, but does not include the subfiles. The file is presented to the client as a single file having the archive file format. A request from the client is received to read the file and the metadata file is read to return data responsive to the request.
1. A system comprising: a processor; and memory configured to store one or more sequences of instructions which, when executed by the processor, cause the processor to carry out the steps of:
receiving, at a deduplication file system, a file from a client for storage in the deduplication file system, the file being in an archive file format that is used by an application on the client, and comprising a plurality of subfiles compressed together in the file according to the archive file format, a plurality of local headers corresponding to the plurality of subfiles, and a central directory used by the application to locate information stored in the file;
decompressing the file to store the plurality of subfiles in the deduplication file system separate from the file;
creating a metadata file describing a structure of the file, the metadata file comprising the plurality of local headers, the central directory, pointers to the plurality of subfiles in the deduplication file system, but not comprising the plurality of subfiles;
presenting the file to the client as a single file having the archive file format;
receiving a request from the client to read the file; and
reading the metadata file to return data responsive to the request, the returned data comprising metadata from the metadata file, file content from one or more of the plurality of subfiles stored separately in the deduplication file system, or both, wherein the plurality of subfiles in the file comprise first and second extensible markup language (XML) files, the first XML, file comprises content inputted by a user of the application, and the second XML, file comprises information generated by the application that describes the content in the first XML file.
2. The system of claim 1 wherein during the presenting, the file has been decompressed from the archive file format, and the plurality of subfiles have been stored separately from each other on the deduplication file system.
3. The system of claim 1 wherein creating the metadata file comprises:
reading the central directory in the file to locate the plurality of local headers in the file, and the plurality of subfiles in the file;
writing the central directory, and the plurality of local headers to the metadata file; and
not writing the plurality of subfiles to the metadata file.
4. The system of claim 1 wherein the processor carries out the step of deduplicating the plurality of subfiles.
5. The system of claim 1 wherein the processor carries out the steps of:
upon receiving the request to read the file, reading the central directory in the metadata file to determine a location of first data responsive to the request;
determining that the location of the first data is in a first subfile, the first subfile being stored separate from the metadata file on the deduplication file system, and the first subfile having been present in the file received from the client; and
reading the first subfile to return the first data to the client.
6. A method comprising:
receiving, at a deduplication file system, a file from a client for storage in the deduplication file system, the file being in an archive file format that is used by an application on the client, and comprising a plurality of subfiles compressed together in the file according to the archive file format, a plurality of local headers corresponding to the plurality of subfiles, and a central directory used by the application to locate information stored in the file;
decompressing the file to store the plurality of subfiles in the deduplication file system separate from the file;
creating a metadata file describing a structure of the file, the metadata file comprising the plurality of local headers, the central directory, pointers to the plurality of subfiles in the deduplication file system, but not comprising the plurality of subfiles;
presenting the file to the client as a single file having the archive file format;
receiving a request from the client to read the file; and
reading the metadata file to return data responsive to the request, the returned data comprising metadata from the metadata file, file content from one or more of the plurality of subfiles stored separately in the deduplication file system, or both, wherein the plurality of subfiles in the file comprise first and second extensible markup language (XML) files, the first XML, file comprises content inputted by a user of the application, and the second XML, file comprises information generated by the application that describes the content in the first XML file.
7. The method of claim 6 wherein during the presenting, the file has been decompressed from the archive file format, and the plurality of subfiles have been stored separately from each other on the deduplication file system.
8. The method of claim 6 wherein creating the metadata file comprises:
reading the central directory in the file to locate the plurality of local headers in the file, and the plurality of subfiles in the file;
writing the central directory, and the plurality of local headers to the metadata file; and
not writing the plurality of subfiles to the metadata file.
9. The method of claim 6 comprising deduplicating the plurality of subfiles.
10. The method of claim 6 comprising:
upon receiving the request to read the file, reading the central directory in the metadata file to determine a location of first data responsive to the request;
determining that the location of the first data is in a first subfile, the first subfile being stored separate from the metadata file on the deduplication file system, and the first subfile having been present in the file received from the client; and
reading the first subfile to return the first data to the client.
11. A computer program product, comprising a non-transitory computer-readable medium having a computer-readable program code embodied therein, the computer-readable program code adapted to be executed by one or more processors to implement a method comprising:
receiving, at a deduplication file system, a file from a client for storage in the deduplication file system, the file being in an archive file format that is used by an application on the client, and comprising a plurality of subfiles compressed together in the file according to the archive file format, a plurality of local headers corresponding to the plurality of subfiles, and a central directory used by the application to locate information stored in the file;
decompressing the file to store the plurality of subfiles in the deduplication file system separate from the file;
creating a metadata file describing a structure of the file, the metadata file comprising the plurality of local headers, the central directory, pointers to the plurality of subfiles in the deduplication file system, but not comprising the plurality of subfiles;
presenting the file to the client as a single file having the archive file format;
receiving a request from the client to read the file; and
reading the metadata file to return data responsive to the request, the returned data comprising metadata from the metadata file, file content from one or more of the plurality of subfiles stored separately in the deduplication file system, or both, wherein the plurality of subfiles in the file comprise first and second extensible markup language (XML) files, the first XML, file comprises content inputted by a user of the application, and the second XML, file comprises information generated by the application that describes the content in the first XML file.
12. The computer program product of claim 11 wherein during the presenting, the file has been decompressed from the archive file format, and the plurality of subfiles have been stored separately from each other on the deduplication file system.
13. The computer program product of claim 11 wherein creating the metadata file comprises:
reading the central directory in the file to locate the plurality of local headers in the file, and the plurality of subfiles in the file;
writing the central directory, and the plurality of local headers to the metadata file; and
not writing the plurality of subfiles to the metadata file.
14. The computer program product of claim 11 wherein the method comprises deduplicating the plurality of subfiles.
15. The computer program product of claim 11 wherein the method comprises:
upon receiving the request to read the file, reading the central directory in the metadata file to determine a location of first data responsive to the request;
determining that the location of the first data is in a first subfile, the first subfile being stored separate from the metadata file on the deduplication file system, and the first subfile having been present in the file received from the client; and
reading the first subfile to return the first data to the client.