Data management in a multi-tenant distributive environment
The present invention relates to a method and apparatus for data management in multi-tenant distributive environment. According to the embodiments of the present invention, a tenant-specific virtual metadata server may be created and registered with a task manager in a job processing system. The virtual metadata server provides a tenant-specific logical file system namespace. Any access to tenant data will be performed via the virtual metadata server. In other words, such virtual metadata server acts as an interface between a tenant terminal and an entity file system. In this way, it may effectively realize isolation of tenant data and enhance data security. Moreover, by using a virtual metadata server rather than physically isolating user data, the embodiments of the present invention may guarantee the efficiency of data sharing while improving the data security.
1. A method for managing tenant data in a multi-tenant distributed environment, the method comprising:
receiving, from a tenant, a request for resources, the request for resources intended for processing a job;
in response to the request, providing a virtual metadata server within a tenant client specific to the tenant, the virtual metadata server providing an independent logical file system namespace for the tenant, wherein the independent logical file system namespace is enabled to isolate the tenant data from other data stored within the multi-tenant distributed environment;
wherein the virtual metadata server is enabled to receive requests to modify which portions of data within the multi-tenant distributed environment are available to the tenant; and
registering the virtual metadata server with a job manager, the job manager configured to access data related to the tenant via the virtual metadata server;
creating an address map associated with the tenant, wherein the address map converts a logical address in the logical file system namespace into an actual address in a global file system namespace of a distributed file system that stores the tenant data.
2. The method according to claim 1 , further comprising:
receiving the logical address for the data from the virtual metadata server;
converting the logical address into the actual address by means of the address map; and
providing the actual address to the job manager via the virtual metadata server.
3. The method according to claim 1 , further comprising:
receiving, from a second tenant, a request for sharing data with the tenant; and
in response to the request received from the second tenant, configuring the virtual metadata server specific to the tenant to add a logical address of to-be-shared data in the logical file system namespace.
4. The method according to claim 3 , further comprising:
adding a mapping entry associated with the to-be-shared data in the address map associated with the tenant, for converting the logical address of the to-be-shared data into an actual address in a global file system namespace of a distributed file system that stores the tenant data.
5. The method according to claim 1 , the method further comprising:
assigning a memory node for storing data associated with the tenant based on a service level agreement (SLA), wherein the request from the tenant contains an indication of the SLA.
6. The method according to claim 5 , further comprising:
in response to the SLA being greater than a predetermined threshold, storing the data associated with the tenant on at least one of a dedicated memory node and on a memory node associated with a higher security level.
7. The method according to claim 1 , wherein data associated with multiple different tenants is stored into a single distributed file system.
8. The method according to claim 1 , wherein providing the virtual metadata server specific to the tenant further comprises:
configuring the virtual metadata server to execute at least a part of a metadata server of a distributed file system that stores the tenant data.
9. A method for managing distributed job processing, the method comprising:
registering a virtual metadata server within a tenant client specific to a tenant, the virtual metadata server providing an independent logical file system namespace for the tenant, wherein the independent logical file system namespace is enabled to isolate the tenant data from other data stored within the multi-tenant distributed environment;
wherein the virtual metadata server is enabled to receive requests to modify which portions of data within the multi-tenant distributed environment are available to the tenant; and
accessing data related to the tenant via the virtual metadata server while processing a job of the tenant
creating an address map associated with the tenant, wherein the address map converts a logical address in the logical file system namespace into an actual address in a global file system namespace of a distributed file system that stores the tenant data.
10. The method according to claim 9 , wherein accessing data related to the tenant via the virtual metadata server further comprises:
receiving, from the tenant, a request for processing the job;
in response to the request, requesting a memory location of the data related to the job with the virtual metadata server; and
receiving, from the virtual metadata server, an actual address of the data related to the job in a global file system namespace of a distributed file system that stores tenant data, wherein the actual address representing the memory location.
11. The method according to claim 10 , further comprising:
assigning an execution node for executing the job of the tenant based on the actual address.
12. An apparatus for managing tenant data in a multi-tenant distributed environment, the apparatus comprising:
a managing unit wherein the managing unit is configured to receive, from a tenant, a request for resources for processing a job;
in response to the request, provide a virtual metadata server within a tenant client specific to the tenant, the virtual metadata server providing an independent logical file system namespace for the tenant wherein the independent logical file system namespace is enabled to isolate the tenant data from other data stored within the multi-tenant distributed environment;
configured to create an address map associated with the tenant, wherein the address mapping converts a logical address in the logical file system namespace into an actual address in a global file system namespace of a distributed file system that stores tenant data;
wherein the virtual metadata server is enabled to receive requests to modify which portions of data within the multi-tenant distributed environment are available to the tenant; and
register the virtual metadata server with a job manager wherein the job manager is configured to accesses data related to the tenant via the virtual metadata server.
13. The apparatus according to claim 12 , further configured to
receive the logical address of the data from the virtual metadata server;
convert the logical address into the actual address by means of the address mapping; and
provide the actual address to the job manager via the virtual metadata server.
14. The apparatus according to claim 12 , further configured to receive, from a second tenant, a request for sharing data with the tenant; and
in response to the request received from the second tenant, configure the virtual metadata server specific to the tenant to add a logical address of to-be-shared data in the logical file system namespace, and adding a mapping entry associated with the to-be-shared data in the address map associated with the tenant, for converting the logical address of the to-be-shared data into an actual address in a global file system namespace of a distributed file system that stores tenant data.
15. The apparatus according to claim 12 , wherein the request from the tenant contains an indication of a service level agreement (SLA), the apparatus configured to assign a memory node for storing data related to the tenant based on the SLA, and in response to the SLA being greater than a predetermined threshold, store the data associated with the tenant on at least one of a dedicated memory node and on a memory node associated with a higher security level.
16. The apparatus according to claim 12 , wherein data associated with multiple different tenants is stored into a single distributed file system.
17. The apparatus according to claim 12 , further configured to configure the virtual metadata server to execute at least a part of a metadata server of a distributed file system that stores tenant data.
18. An apparatus for managing distributed job processing, the apparatus comprising a managing unit configured to register a virtual metadata server within a tenant client specific to a tenant, the virtual metadata server providing an independent logical file system namespace for the tenant;
during processing a job of the tenant, access data related to the tenant via the virtual metadata server, wherein the virtual metadata server provides an independent logical file system namespace enabled to isolate the data from other data stored within the multi-tenant distributed environment;
wherein the virtual metadata server is enabled to receive requests to modify which portions of data within the multi-tenant distributed environment are available to the tenant;
receiving, from the tenant, a request for processing a job; in response to the request, request a memory location of the data related to the job with the virtual metadata server, and receiving, from the virtual metadata server, an actual address of the data related to the job in a global file system namespace of a distributed file system that stores tenant data, the actual address representing the memory location, and assign an execution node for executing the job of the tenant based on the actual address;
creating an address map associated with the tenant, wherein the address map converts a logical address in the logical file system namespace into an actual address in a global file system namespace of a distributed file system that stores the tenant data.