Data Classification Aware Object Storage
Example apparatus and methods process data that is going to be stored in an object store. The object store may have multiple data destinations (e.g., “buckets”). Different buckets have different data storage policies that control, for example, how many copies of the data will be made, whether the data will be stored onsite or offsite, or other storage parameters. Data may be classified by identifying a value for an attribute (e.g., file type, file source) of the data. A storage policy associated with a bucket may then be selected based on the attribute. Once the storage policy has been selected, then the data may be provided to a bucket associated with the storage policy. The number of buckets, data classifications, or storage policies may be updated by adaptive parameterization that considers the amount or type of data observed and stored in the object store.
1 . A non-transitory computer-readable storage medium storing computer-executable instructions that when executed by a computer cause the computer to perform a method, the method comprising:
accessing data that is to be stored in an object store, where the object store is configured with two or more data destinations, where different data destinations have different data storage policies;
classifying the data by identifying a value for an attribute of the data;
selecting a data storage policy associated with a member of the two or more data destinations based, at least in part, on the value of the attribute, and
providing the data to a member of the two or more data destinations that is associated with the data storage policy.
2 . The non-transitory computer-readable storage medium of claim 1 , where classifying the data by identifying the value for the attribute includes examining metadata associated with the data or examining the contents of the data.
3 . The non-transitory computer-readable storage medium of claim 2 , where the attribute is a file type, a file size, a file owner, an origin of the data, an age of the data, or a velocity of the data.
4 . The non-transitory computer-readable storage medium of claim 1 , where the data storage policy describes a number of copies to be made of the data, whether the data is to be stored onsite, whether the data is to be stored offsite, whether the data is to be compressed, a type of compression to be performed on the data, whether the data is to be encrypted, or a type of encryption to be performed on the data.
5 . The non-transitory computer-readable storage medium of claim 1 , where the data storage policy controls whether the data will be stored using erasure codes.
6 . The non-transitory computer-readable storage medium of claim 5 , where the data storage policy controls a parity level used with erasure code based storage.
7 . The non-transitory computer-readable storage medium of claim 1 , where classifying the data is performed in an apparatus located external to the object store, where the object store exposes two or more namespaces to the apparatus located external to the object store, and where members of the two or more data destinations have unique namespaces.
8 . The non-transitory computer-readable storage medium of claim 1 , where classifying the data is performed in an apparatus located internal to the object store, where the object store exposes two or more namespaces to the apparatus located internal to the object store, where the object store exposes a single namespace to apparatus located outside the object store, and where members of the two or more data destinations have unique namespaces.
9 . The non-transitory computer-readable storage medium of claim 1 , where classifying the data is performed in-line in an apparatus integrated into the object store, where the object store exposes two or more namespaces to the apparatus integrated into the object store, where the object store exposes a single namespace to apparatus located outside the object store, and where members of the two or more data destinations have unique namespaces.
10 . The non-transitory computer-readable storage medium of claim 1 , where providing the data to the member includes providing the data directly to the member.
11 . The non-transitory computer-readable storage medium of claim 1 , where providing the data to the member includes sending the data to the member indirectly via an intermediate data store.
12 . The non-transitory computer-readable storage medium of claim 11 , the method comprising controlling a separate process to move the data from the intermediate data store to the member.
13 . The non-transitory computer-readable storage medium of claim 12 , the method comprising:
selectively triggering the separate process periodically or upon determining that a threshold amount of data is being stored in the intermediate data store.
14 . The non-transitory computer-readable storage medium of claim 1 , the method comprising:
selectively adding a new data destination to the two or more data destinations, where adding the new data destination includes providing an additional data storage policy associated with the new data destination, or
selectively removing a data destination from the two or more data destinations, where removing the data destination includes deactivating a data storage policy associated with the data destination being removed.
15 . The non-transitory computer-readable storage medium of claim 1 , the method comprising:
selectively modifying a data storage policy based, at least in part, on an observation of a threshold amount of data that has been stored in the object store.
16 . An apparatus, comprising:
a processor;
a memory;
a set of logics that control storage of data in an object store configured with two or more buckets, where the set of logics cause an item to be stored in a member of the two or more buckets based, at least in part, on a set of data classifications; and
an interface that connects the processor, the memory, and the set of logics;
the set of logics comprising:
a first logic that produces a classification of the item to be stored by the object store;
a second logic that selects a bucket from the two or more buckets based, at least in part, on the classification; and
a third logic that controls the item to be provided to the bucket.
17 . The apparatus of claim 16 , where the first logic produces the classification from the contents of the item or from metadata associated with the item.
18 . The apparatus of claim 17 , where the second logic selects the bucket by matching the classification to storage parameters associated with members of the two or more buckets.
19 . The apparatus of claim 18 , where the third logic controls the item to be provided to the bucket indirectly via a network attached storage.
20 . The apparatus of claim 17 , comprising a fourth logic that:
selectively reconfigures the number of buckets available in the object store upon determining that a threshold number of buckets are being utilized below an under-utilization threshold or upon determining that a threshold number of buckets are being utilized above an over-utilization threshold, and
selectively reconfigures a storage parameter associated with a bucket upon determining that less than a lower threshold amount of data has been stored according to the storage parameter or upon determining that more than an upper threshold amount of data has been stored according to the storage parameter.
21 . An object store, comprising:
means for partitioning an object store into a plurality of data stores, where a member of the plurality of data stores is associated with a unique addressable namespace and a set of storage parameters;
means for dynamically establishing the set of storage parameters for a member of the plurality of data stores;
means for identifying a set of attributes for a file to be stored in a member of the plurality of data stores; and
means for selecting a member of the plurality of data stores to store the file based, at least in part, on a comparison of the set of attributes and the set of storage parameters for the member of the plurality of data stores.