Multi-tier storage system configured for efficient management of small files associated with Internet of Things

1. An apparatus comprising:

a multi-tier storage system comprising at least a front-end storage tier, a back-end storage tier and a data mover module configured to control movement of data between the storage tiers;

the multi-tier storage system being configured to receive a plurality of files;

the front-end storage tier comprising at least one key-value store and a file system comprising at least one specified directory;

wherein the multi-tier storage system is configured to combine multiple ones of the received files written to the at least one specified directory of the file system of the front-end storage tier into an object for storage in association with a key utilized to identify the object in the at least one key-value store;

wherein the object comprises multiple sub-keys identifying respective ones of the multiple files within the object, the multiple sub-keys being sub-keys of the key utilized to identify the object in the at least one key-value store of the front-end storage tier;

wherein the data mover module is configured to copy the object from the at least one key-value store to the at least one specified directory of the front-end storage tier;

wherein the received files are provided to the file system of the front-end storage tier by one or more Internet of Things (IoT) nodes over at least one network, the one or more IoT nodes writing the multiple files each having a file size at or below a specified threshold to said at least one specified directory, said at least one specified directory being designated for files having the file size at or below the specified threshold;

wherein the data mover module comprises a synchronizer service component that accesses the at least one key-value store of the front-end storage tier via a direct memory application programming interface;

wherein the synchronizer service component (i) provides the multiple files to the at least one key-value store via the direct memory application programming interface for combination into the object and (ii) performs at least one of a file collection operation and a compression operation in conjunction with copying of the object from the at least one key-value store to said at least one specified directory; and

wherein the apparatus is implemented using at least one processing device comprising a processor coupled to a memory.

2. The apparatus of claim 1 wherein the received files contain data from respective ones of a plurality of IoT devices coupled to the one or more IoT nodes.

3. The apparatus of claim 1 wherein the object comprises a binary large object and the multiple sub-keys identify respective offsets of the multiple files within the binary large object.

4. The apparatus of claim 1 wherein the data mover module is configured to store the copied object as one or more blocks in the back-end storage tier.

5. The apparatus of claim 4 wherein the file collection operation comprises a tar operation, wherein the data mover module is configured to perform the tar and compression operations in conjunction with storing of the copied object as one or more blocks in the back-end storage tier and wherein the tar and compression operations are applied separately to each of the one or more blocks of the object.

6. The apparatus of claim 4 wherein the back-end storage tier comprises a Hadoop Distributed File System (HDFS) and the object is stored in the back-end storage tier as one or more maximum-size HDFS file blocks.

7. The apparatus of claim 1 wherein the data mover module is configured to identify the multiple files from among the received files as each having the file size at or below the specified threshold and wherein the writing of the multiple files to said at least one specified directory is performed via the data mover module before combining the multiple files into the object.

8. The apparatus of claim 1 wherein the front-end storage tier comprises a fast tier having a relatively high input-output processing speed and a relatively low capacity and the back-end storage tier comprises a capacity tier having a relatively low input-output processing speed and a relatively high capacity.

9. The apparatus of claim 1 wherein the front-end storage tier comprises portions of respective memories of one or more compute nodes or one or more input-output nodes.

10. The apparatus of claim 9 wherein the portions of the respective memories of the one or more compute nodes or the one or more input-output nodes comprise respective flash memories of those nodes.

11. The apparatus of claim 4 wherein the data mover module is configured to retrieve the object from the back-end storage tier into the front-end storage tier with the multiple sub-keys being utilized to access respective ones of the multiple files of the object.

12. The apparatus of claim 1 wherein the object is made accessible to a given analytics application running on a compute node via a corresponding dynamically loadable namespace identifier.

13. An information processing system comprising the apparatus of claim 1 and the one or more IoT nodes each of which communicates with the multi-tier storage system over the at least one network.

14. A method comprising:

receiving a plurality of files in a multi-tier storage system comprising at least a front-end storage tier, a back-end storage tier and a data mover module configured to control movement of data between the storage tiers;

combining multiple ones of the received files written to at least one specified directory of a file system of the front-end storage tier into an object for storage in association with a key utilized to identify the object in a key-value store of the front-end storage tier; and

copying the object from the key-value store to said at least one specified directory of the front-end storage tier;

wherein the object comprises multiple sub-keys identifying respective ones of the multiple files within the object, the multiple sub-keys being sub-keys of the key utilized to identify the object in the key-value store of the front-end storage tier;

wherein the received files are provided to the file system of the front-end storage tier by one or more Internet of Things (IoT) nodes over at least one network, the one or more IoT nodes writing the multiple files each having a file size at or below a specified threshold to said at least one specified directory, said at least one specified directory being designated for files having the file size at or below the specified threshold;

wherein the data mover module comprises a synchronizer service component that accesses the key-value store of the front-end storage tier via a direct memory application programming interface;

wherein the synchronizer service component (i) provides the multiple files to the key-value store via the direct memory application programming interface for combination into the object and (ii) performs at least one of a file collection operation and a compression operation in conjunction with the copying of the object from the key-value store to said at least one specified directory; and

wherein the steps of the method are performed by at least one processing device comprising a processor coupled to a memory.

15. The method of claim 14 further comprising:

storing the copied object as one or more blocks in the back-end storage tier.

16. A non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code when executed by at least one processing device causes said processing device:

to receive a plurality of files in a multi-tier storage system comprising at least a front-end storage tier, a back-end storage tier and a data mover module configured to control movement of data between the storage tiers;

to combine multiple ones of the received files written to at least one specified directory of a file system of the front-end storage tier into an object for storage in association with a key utilized to identify the object in a key-value store of the front-end storage tier; and

to copy the object from the key-value store to said at least one specified directory of the front-end storage tier;

wherein the object comprises multiple sub-keys identifying respective ones of the multiple files within the object, the multiple sub-keys being sub-keys of the key utilized to identify the object in the key-value store of the front-end storage tier;

wherein the received files are provided to the file system of the front-end storage tier by one or more Internet of Things (IoT) nodes over at least one network, the one or more IoT nodes writing the multiple files each having a file size at or below a specified threshold to said at least one specified directory, said at least one specified directory being designated for files having the file size at or below the specified threshold;

wherein the data mover module comprises a synchronizer service component that accesses the key-value store of the front-end storage tier via a direct memory application programming interface; and

wherein the synchronizer service component (i) provides the multiple files to the key-value store via the direct memory application programming interface for combination into the object and (ii) performs at least one of a file collection operation and a compression operation in conjunction with the copying of the object from the key-value store to said at least one specified directory.

17. The processor-readable storage medium of claim 16 wherein the program code when executed by at least one processing device further causes said processing device:

to store the copied object as one or more blocks in the back-end storage tier.

18. The processor-readable storage medium of claim 17 wherein the file collection operation comprises a tar operation, wherein the data mover module is configured to perform the tar and compression operations in conjunction with storing of the copied object as one or more blocks in the back-end storage tier and wherein the tar and compression operations are applied separately to each of the one or more blocks of the object.

19. The method of claim 14 wherein the object comprises a binary large object and the multiple sub-keys identify respective offsets of the multiple files within the binary large object.

20. The method of claim 15 wherein the file collection operation comprises a tar operation, wherein the data mover module is configured to perform the tar and compression operations in conjunction with the storing of the copied object as one or more blocks in the back-end storage tier and wherein the tar and compression operations are applied separately to each of the one or more blocks of the object.