Method, electronic device, and computer program product for storing and accessing data across multiple servers utilizing metadata of a distributed hash table
Embodiments of the present disclosure relate to a method, an electronic device, and a computer program product for storing and accessing data. A method for storing data includes: dividing, in response to receiving a request for storing data from a client, the data into a plurality of data blocks; storing the plurality of data blocks in a plurality of servers respectively; generating metadata of the data to record corresponding addresses for storing the plurality of data blocks in the plurality of servers; and storing the generated metadata in a metadata repository. The embodiments of the present disclosure can effectively improve data transmission efficiency, data availability, and data security in a cloud storage system.
1. A method, comprising:
dividing, in response to receiving a request for storing data from a client, the data into a plurality of data blocks;
storing the plurality of data blocks in a plurality of servers, respectively, wherein the plurality of servers are from respective ones of a plurality of cloud providers;
generating metadata of the data to record corresponding addresses for storing the plurality of data blocks in the plurality of servers; and
storing the generated metadata in a metadata repository;
wherein the metadata repository is implemented at least in part using a distributed hash table for storing the metadata of the plurality of data blocks and check block metadata of at least one check block, the distributed hash table comprising a first entry corresponding to a hash value of a first data block and an address of a first server from a first one of the plurality of cloud providers for storing the first data block, a second entry corresponding to a hash value of a second data block and an address of a second server from a second one of the plurality of cloud providers, different than the first one of the plurality of cloud providers, for storing the second data block, and a third entry corresponding to a hash value of the at least one check block and an address of a third server from a third one of the plurality of cloud providers, different than the first one of the plurality of cloud providers and the second one of the plurality of cloud providers, for storing the at least one check block.
2. The method according to claim 1 , further comprising:
generating the at least one check block for checking the plurality of data blocks;
storing the at least one check block in at least one server of the plurality of servers;
generating the check block metadata of the at least one check block to record at least one address for storing the at least one check block in the at least one server; and
storing the generated check block metadata in the metadata repository.
3. A computer program product tangibly stored in a non-transitory computer-readable storage medium and comprising machine-executable instructions that, when executed by a device, cause the device to perform the method of claim 1 .
4. The computer program product according to claim 3 , wherein the machine-executable instructions, when executed by a device, further cause the device to:
generate the at least one check block for checking the plurality of data blocks;
store the at least one check block in at least one server of the plurality of servers;
generate the check block metadata of the at least one check block to record at least one address for storing the at least one check block in the at least one server; and
store the generated check block metadata in the metadata repository.
5. The method according to claim 1 , wherein the at least one check block is associated with configuration information that indicates a number of data blocks utilized in generating the at least one check block, the configuration information being part of the check block metadata stored in the metadata repository and being separate from the at least one check block.
6. A method, comprising:
acquiring, in response to receiving a request for accessing data from a client, metadata of the data from a metadata repository, the data being divided into a plurality of data blocks, and the metadata recording corresponding addresses for storing the plurality of data blocks in a plurality of servers, wherein the plurality of servers are from respective ones of a plurality of cloud providers;
determining, based on the metadata, the corresponding addresses for storing the plurality of data blocks in the plurality of servers;
acquiring the plurality of data blocks from the corresponding addresses in the plurality of servers; and
combining, in response to the plurality of data blocks being acquired, the plurality of data blocks into the data for the client to access;
wherein the metadata repository is implemented at least in part using a distributed hash table for storing the metadata of the plurality of data blocks and check block metadata of at least one check block, the distributed hash table comprising a first entry corresponding to a hash value of a first data block and an address of a first server from a first one of the plurality of cloud providers for storing the first data block, a second entry corresponding to a hash value of a second data block and an address of a second server from a second one of the plurality of cloud providers, different than the first one of the plurality of cloud providers, for storing the second data block, and a third entry corresponding to a hash value of the at least one check block and an address of a third server from a third one of the plurality of cloud providers, different than the first one of the plurality of cloud providers and the second one of the plurality of cloud providers, for storing the at least one check block.
7. The method according to claim 6 , further comprising:
determining, in response to a failure to acquire at least one data block of the plurality of data blocks, content of the at least one data block based on the at least one check block for checking the plurality of data blocks and other data blocks that have been acquired in the plurality of data blocks; and
combining the plurality of data blocks into the data for the client to access.
8. The method according to claim 7 , further comprising:
acquiring the check block metadata of the at least one check block from the metadata repository, the check block metadata recording at least one address for storing the at least one check block in the plurality of servers;
determining, based on the check block metadata, the at least one address for storing the at least one check block in the plurality of servers; and
acquiring the at least one check block from the at least one address.
9. An electronic device, comprising:
at least one processing unit; and
at least one memory which is coupled to the at least one processing unit and stores instructions for execution by the at least one processing unit, wherein the instructions, when executed by the at least one processing unit, cause the electronic device to perform actions comprising the method of claim 6 .
10. The electronic device according to claim 9 , wherein the actions further comprise:
determining, in response to a failure to acquire at least one data block of the plurality of data blocks, content of the at least one data block using the at least one check block for checking the plurality of data blocks and other data blocks that have been acquired in the plurality of data blocks; and
combining the plurality of data blocks into the data for the client to access.
11. The electronic device according to claim 10 , wherein the actions further comprise:
acquiring the check block metadata of the at least one check block from the metadata repository, the check block metadata recording at least one address for storing the at least one check block in the plurality of servers;
determining, based on the check block metadata, the at least one address for storing the at least one check block in the plurality of servers; and
acquiring the at least one check block from the at least one address.
12. A computer program product tangibly stored in a non-transitory computer-readable storage medium and comprising machine-executable instructions that, when executed by a device, cause the device to perform the method of claim 5 .
13. The computer program product according to claim 6 , wherein the machine-executable instructions, when executed by a device, further cause the device to:
determine, in response to a failure to acquire at least one data block of the plurality of data blocks, content of the at least one data block using the at least one check block for checking the plurality of data blocks and other data blocks that have been acquired in the plurality of data blocks; and
combine the plurality of data blocks into the data for the client to access.
14. The computer program product according to claim 13 , wherein the machine-executable instructions, when executed by a device, further cause the device to:
acquire the check block metadata of the at least one check block from the metadata repository, the check block metadata recording at least one address for storing the at least one check block in the plurality of servers;
determine, based on the check block metadata, the at least one address for storing the at least one check block in the plurality of servers; and
acquire the at least one check block from the at least one address.
15. The method according to claim 6 , wherein the at least one check block is associated with configuration information that indicates a number of data blocks utilized in generating the at least one check block, the configuration information being part of the check block metadata stored in the metadata repository and being separate from the at least one check block.
16. An electronic device, comprising:
at least one processing unit; and
at least one memory which is coupled to the at least one processing unit and stores instructions for execution by the at least one processing unit, wherein the instructions, when executed by the at least one processing unit, cause the electronic device to perform actions comprising:
dividing, in response to receiving a request for storing data from a client, the data into a plurality of data blocks;
storing the plurality of data blocks in a plurality of servers, respectively, wherein the plurality of servers are from respective ones of a plurality of cloud providers;
generating metadata of the data to record corresponding addresses for storing the plurality of data blocks in the plurality of servers; and
storing the generated metadata in a metadata repository;
wherein the metadata repository is implemented at least in part using a distributed hash table for storing the metadata of the plurality of data blocks and check block metadata of at least one check block, the distributed hash table comprising a first entry corresponding to a hash value of a first data block and an address of a first server from a first one of the plurality of cloud providers for storing the first data block, a second entry corresponding to a hash value of a second data block and an address of a second server from a second one of the plurality of cloud providers, different than the first one of the plurality of cloud providers, for storing the second data block, and a third entry corresponding to a hash value of the at least one check block and an address of a third server from a third one of the plurality of cloud providers, different than the first one of the plurality of cloud providers and the second one of the plurality of cloud providers, for storing the at least one check block.
17. The electronic device according to claim 16 , wherein the actions further comprise:
generating the at least one check block for checking the plurality of data blocks;
storing the at least one check block in at least one server of the plurality of servers;
generating the check block metadata of the at least one check block to record at least one address for storing the at least one check block in the at least one server; and
storing the generated check block metadata in the metadata repository.
18. The electronic device according to claim 16 , wherein the at least one check block is associated with configuration information that indicates a number of data blocks utilized in generating the at least one check block, the configuration information being part of the check block metadata stored in the metadata repository and being separate from the at least one check block.