Multi-tier storage system with data mover modules providing distributed multi-part data movement
An apparatus in one embodiment comprises a storage system having at least first and second storage tiers each comprising a plurality of storage devices. The storage system implements a plurality of data mover modules for controlling movement of data objects between the first and second storage tiers. A given one of the data objects is transferred between the first and second storage tiers utilizing a distributed multi-part data movement protocol in which different portions of the given data object are transferred between the first and second storage tiers by different ones of the data mover modules. For example, each of the different portions may be transferred between the first and second storage tiers by its corresponding data mover module as a plurality of distinct non-overlapping parts. In such an arrangement, each of the plurality of distinct non-overlapping parts of each of the different portions may have a corresponding unique tag identifier.
1. An apparatus comprising:
a storage system comprising at least first and second storage tiers each comprising a plurality of storage devices;
the storage system implementing a plurality of data mover modules for controlling movement of data objects between the first and second storage tiers;
wherein a given one of the data objects is transferred between the first and second storage tiers utilizing a distributed multi-part data movement protocol in which different portions of the given data object are transferred between the first and second storage tiers by different ones of the data mover modules;
wherein the distributed multi-part data movement protocol for transfer of the given data object comprises:
a first phase in which a first one of the data mover modules obtains an upload identifier, communicates the upload identifier to other ones of the data mover modules, and identifies to each of the other data mover modules its corresponding one of the different portions of the given data object to be transferred;
a second phase in which the first one of the data mover modules transfers its corresponding one of the different portions of the given data object in a plurality of distinct non-overlapping parts each having a unique tag identifier, and receives from each of the other data mover modules unique tag identifiers of respective ones of a plurality of distinct non-overlapping parts of the corresponding one of the different portions transferred by that data mover module; and
a third phase in which the first one of the data mover modules completes the transfer of the given data object utilizing the unique tag identifiers for all of the distinct non-overlapping parts of each of the different portions.
2. The apparatus of claim 1 wherein the given data object comprises one of a single data file and a plurality of data files.
3. The apparatus of claim 1 wherein the first storage tier implements a parallel file system comprising a plurality of parallel file system nodes.
4. The apparatus of claim 1 wherein the second storage tier implements at least one object store comprising a plurality of object storage nodes and in which stored data objects have respective unique object identifiers.
5. The apparatus of claim 1 wherein each of the different portions is transferred between the first and second storage tiers by its corresponding data mover module as a plurality of distinct non-overlapping parts.
6. The apparatus of claim 5 wherein each of the plurality of distinct non-overlapping parts of each of the different portions has a corresponding unique tag identifier.
7. The apparatus of claim 1 wherein the data mover modules are configurable in a first mode of operation in which a first subset of the data mover modules are utilized in transferring a data object of a particular size between the first and second storage tiers and are further configurable in one or more additional modes of operation in which respective different subsets of the data mover modules are utilized in transferring the data object of the particular size between the first and second storage tiers.
8. The apparatus of claim 7 wherein the first mode of operation utilizes fewer of the data mover modules than each of the one or more additional modes of operation.
9. The apparatus of claim 1 wherein the storage system further comprises at least one configuration file associated with the data mover modules, and wherein the data mover modules are configurable in at least first and second different modes of operation, each involving utilization of a different number of the data mover modules to transfer a data object of a particular size between the first and second storage tiers, by storing respective first and second different sets of information in the configuration file.
10. The apparatus of claim 9 wherein the storage system comprises an application programming interface that permits an external component to control selection between the first and second different modes of operation by directing storage of a particular one of the first and second different sets of information in the configuration file.
11. The apparatus of claim 1 further comprising a plurality of compute nodes wherein at least one of the compute nodes implements at least a portion of the first storage tier of the storage system.
12. The apparatus of claim 1 wherein:
a first one of the portions of the given data object is transferred between the first and second storage tiers by a first one of the data mover modules;
a second one of the portions of the given data object different than the first portion is transferred between the first and second storage tiers by a second one of the data mover modules different than the first data mover module; and
the transfer of the second portion by the second data mover module is performed without utilizing the first data mover module.
13. A method comprising:
configuring a storage system with at least first and second storage tiers each comprising a plurality of storage devices;
implementing in the storage system a plurality of data mover modules for controlling movement of data objects between the first and second storage tiers; and
transferring a given one of the data objects between the first and second storage tiers utilizing a distributed multi-part data movement protocol in which different portions of the given data object are transferred between the first and second storage tiers by different ones of the data mover modules;
wherein the distributed multi-part data movement protocol for transfer of the given data object comprises:
a first phase in which a first one of the data mover modules obtains an upload identifier, communicates the upload identifier to other ones of the data mover modules, and identifies to each of the other data mover modules its corresponding one of the different portions of the given data object to be transferred;
a second phase in which the first one of the data mover modules transfers its corresponding one of the different portions of the given data object in a plurality of distinct non-overlapping parts each having a unique tag identifier, and receives from each of the other data mover modules unique tag identifiers of respective ones of a plurality of distinct non-overlapping parts of the corresponding one of the different portions transferred by that data mover module; and
a third phase in which the first one of the data mover modules completes the transfer of the given data object utilizing the unique tag identifiers for all of the distinct non-overlapping parts of each of the different portions.
14. The method of claim 13 wherein the storage system further comprises at least one configuration file associated with the data mover modules, and wherein the data mover modules are configurable in at least first and second different modes of operation, each involving utilization of a different number of the data mover modules to transfer a data object of a particular size between the first and second storage tiers, by storing respective first and second different sets of information in the configuration file.
15. The method of claim 13 wherein:
a first one of the portions of the given data object is transferred between the first and second storage tiers by a first one of the data mover modules;
a second one of the portions of the given data object different than the first portion is transferred between the first and second storage tiers by a second one of the data mover modules different than the first data mover module; and
the transfer of the second portion by the second data mover module is performed without utilizing the first data mover module.
16. The method of claim 13 wherein the data mover modules are configurable in a first mode of operation in which a first subset of the data mover modules are utilized in transferring a data object of a particular size between the first and second storage tiers and are further configurable in one or more additional modes of operation in which respective different subsets of the data mover modules are utilized in transferring the data object of the particular size between the first and second storage tiers.
17. A non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code when executed by a processing platform causes said processing platform:
to configure a storage system with at least first and second storage tiers each comprising a plurality of storage devices;
to implement in the storage system a plurality of data mover modules for controlling movement of data objects between the first and second storage tiers; and
to transfer a given one of the data objects between the first and second storage tiers utilizing a distributed multi-part data movement protocol in which different portions of the given data object are transferred between the first and second storage tiers by different ones of the data mover modules;
wherein the distributed multi-part data movement protocol for transfer of the given data object comprises:
a first phase in which a first one of the data mover modules obtains an upload identifier, communicates the upload identifier to other ones of the data mover modules, and identifies to each of the other data mover modules its corresponding one of the different portions of the given data object to be transferred;
a second phase in which the first one of the data mover modules transfers its corresponding one of the different portions of the given data object in a plurality of distinct non-overlapping parts each having a unique tag identifier, and receives from each of the other data mover modules unique tag identifiers of respective ones of a plurality of distinct non-overlapping parts of the corresponding one of the different portions transferred by that data mover module; and
a third phase in which the first one of the data mover modules completes the transfer of the given data object utilizing the unique tag identifiers for all of the distinct non-overlapping parts of each of the different portions.
18. The processor-readable storage medium of claim 17 wherein the storage system further comprises at least one configuration file associated with the data mover modules, and wherein the data mover modules are configurable in at least first and second different modes of operation, each involving utilization of a different number of the data mover modules to transfer a data object of a particular size between the first and second storage tiers, by storing respective first and second different sets of information in the configuration file.
19. The processor-readable storage medium of claim 17 wherein:
a first one of the portions of the given data object is transferred between the first and second storage tiers by a first one of the data mover modules;
a second one of the portions of the given data object different than the first portion is transferred between the first and second storage tiers by a second one of the data mover modules different than the first data mover module; and
the transfer of the second portion by the second data mover module is performed without utilizing the first data mover module.
20. The processor-readable storage medium of claim 17 wherein the data mover modules are configurable in a first mode of operation in which a first subset of the data mover modules are utilized in transferring a data object of a particular size between the first and second storage tiers and are further configurable in one or more additional modes of operation in which respective different subsets of the data mover modules are utilized in transferring the data object of the particular size between the first and second storage tiers.