Hybrid data management system and method for managing large, varying datasets
A hybrid data management/storage system is provided which includes two or more integrated or connected data management systems. An external application and/or user interacts with the hybrid data management/storage system using a unified interface. Incoming raw data may be directed to be stored in any of a plurality of data management systems based on the incoming data object having one or more of a number of predefined characteristics, including for example size and/or data type. Metadata corresponding to all incoming data objects may be stored in a particular data store, regardless of whether the incoming object's raw data is stored in a different one of the plurality of data stores.
1. A hybrid data storage management system for storing an incoming data object including metadata having first preferred predefined characteristics and raw data having second preferred predefined characteristics, the system comprising:
a plurality of data stores including at least a first data store and a second data store different from the first data store, wherein each of the plurality of data stores is associated with a preferred data store type corresponding to a type of data store whose storage method permits the associated data store to operate more efficiently on data having preferred predefined characteristics associated with the data store type than on data not having the preferred predefined characteristics, and wherein the first data store is a first preferred data store having a first preferred data store type corresponding to first preferred predefined characteristics, and the second data store is a second preferred data store having a second preferred data store type corresponding to second preferred predefined characteristics; and
a routing layer coupled to the plurality of data stores, wherein the routing layer is configured to:
receive, from an external source a write request for the incoming data object;
determine that the metadata and the raw data of the incoming data object have the first and second preferred predefined characteristics, respectively;
identify the first and second preferred data store types corresponding to the metadata and the raw data, respectively, based on the first and second preferred predefined characteristics;
route the raw data to the second data store for storage therein based on the raw data having been identified as corresponding to the second data store type; and
store the metadata in the first data store based on the metadata having been identified as corresponding to the first data store type, and without regard to which of the plurality of data stores is the second preferred data store corresponding to the raw data of the incoming data object, the metadata including associative information linking the metadata with the corresponding raw data in the second data store,
wherein the first data store and the second data store utilize different storage methods, such that, by virtue of a first data store storage method, the first data store operates more efficiently on the metadata than the second data store would, and, by virtue of a second data store storage method, the second data store operates more efficiently on the raw data than the first data store would.
2. The system according to claim 1 , wherein a data management logic operates on the raw data in the second data store in response to detecting an attempted operation on the metadata in the first data store.
3. The system according to claim 2 , wherein the attempted operation corresponds to one of moving, deleting, modifying and copying the incoming data object.
4. The system according to claim 1 , wherein the external source is an external application or user, and the write request is a PUT object operation.
5. The system according to claim 1 , wherein the predefined characteristics include a predefined data object size.
6. The system according to claim 1 , wherein the predefined characteristics include a predefined data type.
7. The system according to claim 1 , wherein said associative information comprises information indicating a location within the second data store where the corresponding raw data is stored.
8. The system according to claim 1 , further comprising data management logic configured to:
receive, from an external application or user, a read request for a previously-stored data object;
retrieve metadata stored in the first data store corresponding to the previously-stored data object, wherein such metadata indicates a location within the second data store where the corresponding raw data is stored; and
read the previously-stored data object from the location indicated by the metadata.
9. The system according to claim 1 , further comprising a data management logic configured to utilize data management functions of the first data store for operating on the incoming data object stored in the second data store.
10. The system according to claim 9 , wherein the routing layer is further configured to:
determine that utilizing a particular data management function of the first data store will have a negative performance effect on the hybrid data management system, and
in response, bypass the particular data management function of the first data store to use a corresponding data management function of the second data store to avoid the negative performance effect.