Method and apparatus for providing access to data objects within another data object
Techniques for providing access to data objects within another data object are described herein. In one embodiment, a compound object including multiple data objects is received and metadata is extracted for a data object from the compound object, where the metadata includes a layout of the data object in view of the compound object. Subsequently, access to one or more of the data objects within the compound object is provided based on the extracted metadata without using an application associated with the compound object. Other methods and apparatuses are also described.
1. A computer-implemented method for managing data objects, comprising:
storing a compound object in a storage, the compound object being a binary file image having a plurality of data objects embedded therein, wherein the compound object is a compressed backup file and the storage is a backup storage;
extracting metadata for each of the plurality of data objects from the compound object including
identifying an unpack tool associated with the compound object based on a type of the compound object, wherein the identified unpack tool is selected from a plurality of unpack tools corresponding to a plurality of different types of compound objects,
executing the identified unpack tool on the compound object to extract the metadata for each data object packed within the compound object without creating an instance of the data object from the compound object, wherein the metadata includes a layout of each data object in view of the compound object, including a size of each data object and a location of each data object within the compound object, the location representing an offset from beginning of the binary file image of the compound object, wherein the metadata further includes an access control list (ACL) attribute of a corresponding one of the data objects, wherein the metadata allows all data objects of the plurality of data objects to be accessed directly from the compound object without an instance of each data object being created;
storing the extracted metadata in a metadata store that is separated from a storage location storing the compound object;
exporting the metadata of the data objects in a file system namespace of a file system of the storage based on the metadata, such that each of the data objects is presented by the file system as if the corresponding data object were stored as a separate physical file in the storage when the data object is individually accessed;
subsequently, in response to a request for accessing a first data object within the compound object, determining a size and a location of the first data object within the compound object based on the metadata associated with the first data object stored the metadata store; and
providing access to the first data object directly from the compound object at the determined location within the compound object without using an application that understands a format of the compound object.
2. The method of claim 1 , wherein the access to the first data object within the compound object is provided via a standard file system call of a file system that manages files stored in the storage, including the compound object and the metadata store.
3. The method of claim 1 , wherein providing access to the first data object within the compound object comprises:
extracting an image of the first data object having the determined size from the determined location of the compound object, wherein the determined location represents an offset from the beginning of the compound object; and
transmitting the image of the first data object to a client of the request without understanding content of the first data object.
4. The method of claim 1 , further comprising exporting the first data object in a namespace of a file system using the extracted metadata associated with the first data object, such that the first data object is accessible via a standard system call of the file system.
5. The method of claim 4 , wherein the compound object is an email mailbox and data objects included therein are individual emails.
6. The method of claim 5 , further comprising collecting input data and output data of the identified unpack tool operating on the compound object.
7. The method of claim 6 further comprising extracting the metadata from the collected input data and output data of the identified unpack tool operating on the compound object.
8. The method of claim 7 , further comprising, in response to an operation of the identified unpack tool, capturing an offset of the data object within the compound object, an identifier of the data object, and a size of the data object as a part of the metadata.
9. The method of claim 7 , further comprising capturing an access control attribute of the data object during the write operation, wherein the subsequent access is provided dependent upon the captured access control attribute of the data object.
10. An apparatus for managing data objects, comprising:
a processor;
a storage device to store a compound object, the compound object being a binary file image having a plurality of data objects embedded therein, wherein the compound object is a compressed backup file and the storage is a backup storage; and
memory to store instructions, which when executed from the memory, cause the processor to
execute a match-up engine in the memory to extract metadata for each of the data objects from the compound object including
identifying an unpack tool associated with the compound object based on a type of the compound object, wherein the identified unpack tool is selected from a plurality of unpack tools corresponding to a plurality of different types of compound objects,
executing the identified unpack tool on the compound object to extract the metadata for the data object packed within the compound object without creating an instance of the data object from the compound object, wherein the metadata includes a layout of the data object in view of the compound object, including a size of each data object and a location of each data object within the compound object, the location representing an offset from beginning of the binary file image of the compound object, wherein the metadata further includes an access control list (ACL) attribute of a corresponding one of the data objects, wherein the metadata allows all data objects of the plurality of data objects to be accessed directly from the compound object without an instance of each data object being created,
store the metadata of the compound object in a metadata store maintained within the storage device, wherein the metadata is stored in a separate storage location from the compound object, wherein the metadata of the data objects is exported in a file system namespace of a file system of the storage based on the metadata, such that each of the data objects is presented by the file system as if the corresponding data object were stored as a separate physical file in the storage when the data object is individually accessed, and
execute an object portal in the memory subsequently, in response to a request for accessing a first data object within the compound object, to determine a size and a location of the first data object within the compound object based on the metadata of the first data object obtained from the metadata store and to provide access to the first data object directly from the compound object at the determined location within the compound object without using an application that understands a format of the compound object.
11. The apparatus of claim 10 , further comprising a file system executed in the memory to manage files stored in the storage including the compound object and the metadata store, wherein the object portal is configured to provide the access to the first data object via a standard file system call of the file system.
12. The apparatus of claim 10 , wherein the metadata store is incorporated into a file directory of a file system, such that the first data object is accessible via a standard system call of the file system.
13. The apparatus of claim 12 , wherein the compound object comprises an email mailbox and the plurality of data objects within the compound object comprises individual emails.
14. The apparatus of claim 13 , further comprising:
a read filter executed in the memory by the processor to capture the input data when the unpack tool reads from the compound object, including a location being read within the compound object; and
a write filter executed in the memory by the processor to capture the output data when the unpack tool attempts to create a new instance of each data object, including an identifier of the data object.
15. The apparatus of claim 14 , wherein the write filter further captures an access control attribute when the unpack tool attempts to set the access control attribute to the new instance of the data object, and wherein the object portal is configured to apply the access control to the first data object.
16. A non-transitory machine-readable medium having computer instructions stored therein, which when executed by a processor, cause the processor to perform a method for managing data objects, the method comprising:
storing a compound object in a storage, the compound object being a binary file image having a plurality of data objects embedded therein, wherein the compound object is a compressed backup file and the storage is a backup storage;
extracting metadata for each of the data objects from the compound object, including
identifying an unpack tool associated with the compound object based on a type of the compound object, wherein the identified unpack tool is selected from a plurality of unpack tools corresponding to a plurality of different types of compound objects,
executing the identified unpack tool on the compound object to extract the metadata for each data object packed within the compound object without creating an instance of the data object from the compound object, wherein the metadata includes a layout of each data object in view of the compound object, including a size of each data object and a location of each data object within the compound object, the location representing an offset from beginning of the binary file image of the compound object, wherein the metadata further includes an access control list (ACL) attribute of a corresponding one of the data objects, wherein the metadata allows all data objects of the plurality of data objects to be accessed directly from the compound object without an instance of each data object being created;
storing the extracted metadata in a metadata store that is separated from a storage location storing the compound object;
exporting the metadata of the data objects in a file system namespace of a file system of the storage based on the metadata, such that each of the data objects is presented by the file system as if the corresponding data object were stored as a separate physical file in the storage when the data object is individually accessed;
subsequently, in response to a request for accessing a first data object within the compound object, determining a size and a location of the first data object within the compound object based on the metadata associated with the first data object stored in the metadata store; and
providing access to the first data object directly from the compound object at the determined location within the compound object without using an application that understands a format of the compound object.
17. The non-transitory machine-readable medium of claim 16 , wherein the access to the first data object within the compound object is provided via a standard file system call of a file system that manages files stored in the storage, including the compound object and the metadata store.
18. The non-transitory machine-readable medium of claim 16 , wherein the method further comprises exporting the first data object in a namespace of a file system using the extracted metadata associated with the first data object, such that the first data object is accessible via a standard system call of the file system.