Intelligent data storage and management for cloud computing
Systems and methods are provided for implementing an intelligent data management system for data storage and data management in a cloud computing environment. For example, a system includes an application server, a distributed data storage system, and an intelligent data management system. The application server is configured to host a data processing application. The distributed data storage system is configured to store data generated by a network of devices associated with the data processing application. The intelligent data management system is configured to manage data storage operations for storing the data generated by the network of devices in the distributed data storage system. For example, the intelligent data management system is configured to determine one or more data types of the data generated by the network of devices and select one of a plurality of repositories within the distributed data storage system to store the data based on the determined data types.
1. A system, comprising:
an application server configured to host at least one data processing application;
a distributed data storage system comprising a hierarchical storage system, wherein the hierarchical storage system comprises a tiered system memory comprising at least a first memory tier comprising volatile memory, and a second memory tier comprising non-volatile memory, and a tiered storage system comprising at least a first storage tier and a second storage tier, wherein at least one of the first and second memory tiers and at least one of the first and second storage tiers comprise respective repositories that are configured to store data that is (i) generated by devices of at least one device network, and (ii) processed by the at least one data processing application; and
an intelligent data management system configured to manage data storage operations for storing the data generated by the devices of the at least one device network in the distributed data storage system;
wherein the intelligent data management system is configured to:
receive data generated by a given device of the at least one device network;
determine one or more data types of the received data;
determine a priority level of the given device based on at least one of a device type of the given device, and an application type of the at least one data processing application which processes the data generated by the given device; and
select at least one of the repositories within at least one of the first and second memory tiers and the first and second storage tiers of the hierarchical storage system to store the received data based at least in part on the determined one or more data types of the received data and the determined priority level of the given device; and
wherein the intelligent data management system is further configured to:
determine data write patterns and data read patterns of the devices of the at least one device network; and
utilize the determined data write patterns and data read patterns to store the data of the devices in locations within the distributed data storage system to increase a throughput of data read and data write operations for the devices of the at least one device network.
2. The system of claim 1 , wherein the respective repositories comprise a file repository and a data repository, and wherein the intelligent data management system is configured to determine unstructured data types and structured data types, and store data which is determined to have an unstructured data type in the file repository and store data which is determined to have a structured data type in the data repository.
3. The system of claim 2 , wherein the data repository comprises one or more of a relational database and a non-relational database.
4. The system of claim 1 , wherein the intelligent data management system is configured to determine device types of the devices of the at least one device network, and assign respective priority levels to the devices based at least in part on the respective determined device types of the devices and the application type of the at least one data processing application.
5. The system of claim 1 , wherein the intelligent data management system is configured to allocate one or more virtual machines or application servers to the devices of the at least one device network based on respective device types of the devices.
6. The system of claim 1 , wherein the intelligent data management system is configured to perform one or more pre-processing operations on the data generated by the devices of the at least one device network before storing the data in the distributed data storage system.
7. The system of claim 1 , wherein the system comprises an Internet-of-Things cloud computing system.
8. The system of claim 1 , wherein the devices of the at least one device network comprise sensor devices which generate sensor data.
9. A method comprising:
configuring an application server to host at least one data processing application;
configuring a distributed data storage system to comprise a hierarchical storage system, wherein the hierarchical storage system comprises a tiered system memory comprising at least a first memory tier comprising volatile memory, and a second memory tier comprising non-volatile memory, and a tiered storage system comprising at least a first storage tier and a second storage tier, wherein at least one of the first and second memory tiers and at least one of the first and second storage tiers comprise respective repositories that are configured to store data that is (i) generated by devices of at least one device network, and (ii) processed by the at least one data processing application;
configuring an intelligent data management system to manage data storage operations for storing the data generated by the devices of the at least one device network in the distributed data storage system by performing a process which comprises:
receiving, by the intelligent data management system, data generated by a given device of the at least one device network;
determining, by the intelligent data management system, one or more data types of the received data;
determining, by the intelligent data management system, a priority level of the given device based on at least one of a device type of the given device, and an application type of the at least one data processing application which processes the data generated by the given device; and
selecting, by the intelligent data management system, at least one of the repositories within at least one of the first and second memory tiers and the first and second storage tiers of the hierarchical storage system to store the received data based at least in part on the determined one or more data types of the received data and the determined priority level of the given device; and
configuring the intelligent data management system to perform a process which comprises:
determining, by the intelligent data management system, data write patterns and data read patterns of the devices of the at least one device network; and
utilizing, by the intelligent data management system, the determined data write patterns and data read patterns to store the data of the devices in locations within the distributed data storage system to increase a throughput of data read and data write operations for the devices of the at least one device network.
10. The method of claim 9 , wherein the respective repositories comprise a file repository and a data repository, wherein determining one or more data types of the data generated by the network of devices comprises determining unstructured data types and structured data types.
11. The method of claim 10 , comprising:
storing data which is determined to have an unstructured data type in the file repository; and
storing data which is determined to have a structured data type in the data repository.
12. The method of claim 10 , wherein the data repository comprises one or more of a relational database and a non-relational database.
13. The method of claim 9 , further comprising:
determining, by the intelligent data management system, device types of the devices of the at least one device network; and
assigning, by the intelligent data management system, respective priority levels to the devices based at least in part on the respective determined device types of the devices and the application type of the at least one data processing application.
14. The method of claim 9 , further comprising allocating, by the intelligent data management system, one or more virtual machines or application servers to the devices of the at least one device network based on respective device types of the devices.
15. The method of claim 9 , further comprising performing, by the intelligent data management system, one or more pre-processing operations on the data generated by the devices of the at least one device network before storing the data in the distributed data storage system.
16. An article of manufacture comprising a non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code is executable by one or more processors to:
configure an application server to host at least one data processing application;
configure a distributed data storage system to comprise a hierarchical storage system, wherein the hierarchical storage system comprises a tiered system memory comprising at least a first memory tier comprising volatile memory, and a second memory tier comprising non-volatile memory, and a tiered storage system comprising at least a first storage tier and a second storage tier, wherein at least one of the first and second memory tiers and at least one of the first and second storage tiers comprise respective repositories that are configured to store data that is (i) generated by devices of at least one device network, and (ii) processed by the at least one data processing application; and
configure an intelligent data management system to manage data storage operations for storing the data generated by the devices of the at least one device network in the distributed data storage system, wherein the intelligent data management system is configured to:
receive data generated by a given device of the at least one device network;
determine one or more data types of the received data;
determine a priority level of the given device based on at least one of a device type of the given device, and an application type of the at least one data processing application which processes the data generated by the given device; and
select at least one of the repositories within at least one of the first and second memory tiers and the first and second storage tiers of the hierarchical storage system to store the received data based at least in part on the determined one or more data types of the received data and the determined priority level of the given device; and
wherein the intelligent data management system is further configured to:
determine data write patterns and data read patterns of the devices of the at least one device network; and
utilize the determined data write patterns and data read patterns to store the data of the devices in locations within the distributed data storage system to increase a throughput of data read and data write operations for the devices of the at least one device network.
17. The article of manufacture of claim 16 , wherein the respective repositories comprise a file repository and a data repository, wherein the intelligent data management system is configured to:
determine unstructured data types and structured data types;
store data which is determined to have an unstructured data type in the file repository; and
store data which is determined to have a structured data type in the data repository.
18. The article of manufacture of claim 16 , wherein the program code is further executable by the one or more processors to configure the intelligent data management system to:
determine device types of the devices of the at least one device network; and
assign respective priority levels to the devices based at least in part on the respective determined device types of the devices and the application type of the at least one data processing application.
19. The method of claim 9 , wherein the at least one device network comprises an Internet-of-Things network, and the devices of the at least one device network comprise sensor devices which generate sensor data.
20. The article of manufacture of claim 16 , wherein the at least one device network comprises an Internet-of-Things network, and the devices of the at least one device network comprise sensor devices which generate sensor data.