Method, device, and computer program product for data management
A method, a device, and a computer program product for data management are provided in embodiments of the present disclosure. A method for data management comprises: acquiring, from a storage system, a torrent file associated with downloading of a target file, the target file being stored in the storage system, and the torrent file comprising at least a plurality of data digests of a plurality of data blocks of the target file; generating metadata for the plurality of data blocks based on the torrent file, the metadata being in a format supported by a unified management system, and the unified management system being configured for data access across the storage system and at least one other storage system; and storing the metadata for data-block-level access to the plurality of data blocks through the unified management system.
1. A method, comprising:
acquiring, from an object storage system having a target file stored thereon in a manner accessible in a file-level granularity, a first set of metadata associated with downloading of the target file via a peer-to-peer protocol, wherein the first set of metadata comprises a plurality of data digests corresponding to a plurality of data blocks into which the target file is logically dividable for a multi-source transfer to a client device in accordance with the peer-to-peer protocol;
generating a second set of metadata based on the first set of metadata, the second set of metadata being in a format supported by a unified management system configured to provide a unified interface for access by one or more client devices to multiple distinct storage systems including the object storage system;
storing the second set of metadata in the unified management system to enable access to portions of the target file in a block-level granularity by the one or more client devices; and
receiving a request from one of the one or more client devices to select individual ones of the plurality of data blocks through the unified management system.
2. The method of claim 1 , wherein generating the second set of metadata comprises:
extracting the plurality of data digests from the first set of metadata;
determining a plurality of storage positions of the plurality of data blocks in the object storage system at least based on the plurality of data digests; and
generating additional metadata to indicate mapping of the plurality of data digests to the plurality of storage positions.
3. The method of claim 2 , wherein generating the second set of metadata further comprises:
creating a hierarchical structure based on the plurality of data digests, a plurality of leaf nodes of the hierarchical structure indicating the plurality of data digests respectively, and a parent node of the plurality of leaf nodes indicating an additional data digest generated based on at least two of the plurality of data digests.
4. The method of claim 3 , wherein the plurality of data digests comprise hash values of the plurality of data blocks, and wherein the hierarchical structure comprises a hash tree.
5. The method of claim 3 , wherein generating the second set of metadata further comprises:
extracting file identification information of the target file from the first set of metadata; and
determining a root node of the hierarchical structure based on the file identification information to map a data digest indicated by the root node to a storage position of the file identification information.
6. The method of claim 1 , wherein acquiring the first set of metadata comprises:
initiating a request for the first set of metadata of the target file to the object storage system through the unified management system;
wherein the first set of metadata comprises a torrent file.
7. The method of claim 1 , wherein a first set of one or more data blocks of the target file are stored in the object storage system and a second set of one or more data blocks are stored in at least one other storage system of the multiple distinct storage systems, the second set of the one or more data blocks being different than the first set of the one or more data blocks.
8. An electronic device, comprising:
a processor; and
a memory coupled to the processor and having instructions stored therein, wherein when executed by the processor, the instructions cause the device to perform actions comprising:
acquiring, from an object storage system having a target file stored thereon in a manner accessible in a file-level granularity, a first set of metadata associated with downloading of the target file via a peer-to-peer protocol, wherein the first set of metadata comprises a plurality of data digests corresponding to a plurality of data blocks into which the target file is logically dividable for a multi-source transfer to a client device in accordance with the peer-to-peer protocol;
generating a second set of metadata based on the first set of metadata, the second set of metadata being in a format supported by a unified management system configured to provide a unified interface for access by one or more client devices to multiple distinct storage systems including the object storage system;
storing the second set of metadata in the unified management system to enable access to portions of the target file in a block-level granularity by the one or more client devices; and
receiving a request from one of the one or more client devices to select individual ones of the plurality of data blocks through the unified management system.
9. The electronic device of claim 8 , wherein generating the second set of metadata comprises:
extracting the plurality of data digests from the first set of metadata;
determining a plurality of storage positions of the plurality of data blocks in the object storage system at least based on the plurality of data digests; and
generating additional metadata to indicate mapping of the plurality of data digests to the plurality of storage positions.
10. The electronic device of claim 9 , wherein generating the second set of metadata further comprises:
creating a hierarchical structure based on the plurality of data digests, a plurality of leaf nodes of the hierarchical structure indicating the plurality of data digests respectively, and a parent node of the plurality of leaf nodes indicating an additional data digest generated based on at least two of the plurality of data digests.
11. The electronic device of claim 10 , wherein the plurality of data digests comprise hash values of the plurality of data blocks, and wherein the hierarchical structure comprises a hash tree.
12. The electronic device of claim 10 , wherein generating the second set of metadata further comprises:
extracting file identification information of the target file from the first set of metadata; and
determining a root node of the hierarchical structure based on the file identification information to map a data digest indicated by the root node to a storage position of the file identification information.
13. The electronic device of claim 8 , wherein acquiring the first set of metadata comprises:
initiating a request for the first set of metadata of the target file to the object storage system through the unified management system;
wherein the first set of metadata comprises a torrent file.
14. The electronic device of claim 8 , wherein a first set of one or more data blocks of the target file are stored in the object storage system and a second set of one or more data blocks are stored in at least one other storage system of the multiple distinct storage systems, the second set of the one or more data blocks being different than the first set of the one or more data blocks.
15. A computer program product, tangibly stored in a non-transitory computer-readable medium and comprising computer-executable instructions, wherein when executed, the computer-executable instructions cause a device to:
acquire, from an object storage system having a target file stored thereon in a manner accessible in a file-level granularity, a first set of metadata associated with downloading of the target file via a peer-to-peer protocol, wherein the first set of metadata comprises a plurality of data digests corresponding to a plurality of data blocks into which the target file is logically dividable for a multi-source transfer to a client device in accordance with the peer-to-peer protocol;
generate a second set of metadata based on the first set of metadata, the second set of metadata being in a format supported by a unified management system configured to provide a unified interface for access by one or more client devices to multiple distinct storage systems including the object storage system;
store the second set of metadata in the unified management system to enable access to portions of the target file in a block-level granularity by the one or more client devices; and
receive a request from one of the one or more client devices to select individual ones of the plurality of data blocks through the unified management system.
16. The computer program product of claim 15 , wherein generating the second set of metadata comprises:
extracting the plurality of data digests from the first set of metadata;
determining a plurality of storage positions of the plurality of data blocks in the object storage system at least based on the plurality of data digests; and
generating additional metadata to indicate mapping of the plurality of data digests to the plurality of storage positions.
17. The computer program product of claim 16 , wherein generating the second set of metadata further comprises:
creating a hierarchical structure based on the plurality of data digests, a plurality of leaf nodes of the hierarchical structure indicating the plurality of data digests respectively, and a parent node of the plurality of leaf nodes indicating an additional data digest generated based on at least two of the plurality of data digests.
18. The computer program product of claim 17 , wherein the plurality of data digests comprise hash values of the plurality of data blocks, and wherein the hierarchical structure comprises a hash tree.
19. The computer program product of claim 17 , wherein generating the second set of metadata further comprises:
extracting file identification information of the target file from the first set of metadata; and
determining a root node of the hierarchical structure based on the file identification information to map a data digest indicated by the root node to a storage position of the file identification information.
20. The computer program product of claim 17 , wherein acquiring the first set of metadata comprises:
initiating a request for the first set of metadata of the target file to the object storage system through the unified management system;
wherein the first set of metadata comprises a torrent file.