Storing a variable number of instances of data objects
Described are systems and methods for storing a variable number of instances of data objects (e.g., 1, 2, 3, or up to N−1 instances, where N is the number of instances of the data object included in primary data) in secondary storage across a data storage network. In some examples, a system for storing a variable number of instances of data objects includes, one or more computing devices storing a set of data objects and multiple storage devices distinct from the one or more computing devices. Each of the multiple storage devices is configured to store at least a single instance of a data object. The system also includes a database configured to store information associated with the data objects. This information includes substantially unique identifiers for the data objects and, for each of the data objects, a number of instances of the data object stored on the multiple storage devices.
1. A system for storing multiple instances of a data object, the system comprising:
one or more non-transitory, computer-readable medium storing computer-executable instructions; and,
one or more processors that, having executed the computer-executable instructions, configure the system to perform a plurality of operations comprising:
determining that one or more data objects of a plurality of data objects are subject to a storage policy, wherein the storage policy indicates a number of instances that a particular data object is to be stored in one or more secondary storage devices;
for each data object determined to be subject to the storage policy:
determining a number of instances of each data object to store in one or more secondary storage devices, wherein:
the number of instances of each data object is determined based on the storage policy, and
the number of instances of each data object is greater than or equal to 2;
for each instance of a data object:
determining a secondary storage device selected from a plurality of secondary storage devices in which to store the instance of the data object, wherein determining the secondary storage device in which to store the instance of the data object is based on one or more criterion comprising:
an available storage capacity of the secondary storage device;
a time to store data on the secondary storage device;
a throughput of storage operations performed on the secondary storage device; and,
a time to access stored on the secondary storage device; and,
storing the instance of the data object in the determined secondary storage device;
receiving request to age the data object, wherein a number of instances of the data object stored in one or more secondary storage devices is equal to or fewer than the determined number of instances, wherein the instances of data object in a first tier of storage; and
aging the data object, w herein the aging comprises moving at least one instance of the data object from the first tier of storage to a second tier of storage, w herein the second tier of storage is associated with an inferior performance compared to the first tier of storage.
2. The system of claim 1 , wherein the plurality of operations further comprises:
generating a unique identifier for the data object;
performing a database lookup of the data object based on the unique identifier to determine a number of instances of the data object that have already been stored; and,
wherein determining the number of instances of each data object to store in the one or more secondary storage devices is further based on the number of instances of the data object that have already been stored.
3. The system of claim 1 , wherein each instance of the data object is stored in a different secondary storage device selected from the plurality of secondary storage devices.
4. The system of claim 1 , wherein determining the secondary storage device in which to store the instance of the data object is based on the storage policy to which the data object is subject.
5. The system of claim 1 , wherein the plurality of operations further comprises:
modifying the determined number of instances of a data object selected from the plurality of data objects based on the data object being common to a predetermined number of client computing devices; and,
storing the modified number of instances of the data object in the one or more secondary storage devices selected from the plurality of secondary storage devices.
6. The system of claim 1 , wherein the plurality of operations further comprises:
storing one or more additional instances of the data object in the one or more secondary storage devices to satisfy the determined number of instances.
7. A method for storing multiple instances of a data object, the method comprising:
determining that one or more data objects of a plurality of data objects are subject to the storage policy, wherein the storage policy indicates a number of instances that a particular data object is to be stored in one or more secondary storage devices;
for each data object determined to be subject to the storage policy:
determining a number of instances of each data object to store in one or more secondary storage devices selected from a plurality of secondary storage devices, wherein:
the number of instances of each data object is determined based on the storage policy, and
the number of instances of each data object is greater than or equal to 2;
for each instance of a data object:
determining a secondary storage device in which to store the instance of the data object, wherein determining the secondary storage device in which to store the instance of the data object is based on one or more criterion comprising:
an available storage capacity of the secondary storage device;
a time to store data on the secondary storage device;
a throughout of storage operations performed on the secondary storage device; and,
a time to access stored on the secondary storage device;
storing the instance of the data object in the determined secondary storage device;
receiving a request to age the data object, wherein the number of instances of the data object stored in one or more secondary storage devices is equal to or fewer than the determined number of instances, wherein the instances of data object in a first tier of storage; and
aging the data object, w herein the aging comprises moving at least one instance of the data object from the first tier of storage to a second tier of storage, wherein the second tier of storage is associated with an inferior performance compared to the first tier of storage.
8. The method of claim 7 , further comprising:
generating a unique identifier for the data object;
performing a database lookup of the data object based on the unique identifier to determine a number of instances of the data object that have already been stored; and,
wherein determining the number of instances of each data object to store in the one or more secondary storage devices is further based on the number of instances of the data object that have already been stored.
9. The method of claim 7 , wherein each instance of the data object is stored in a different secondary storage device selected from the plurality of secondary storage devices.
10. The method of claim 7 , wherein determining the secondary storage device in which to store the instance of the data object is based on the storage policy to which the data object is subject.
11. The method of claim 7 , further comprising:
modifying the determined number of instances of a data object selected from the plurality of data objects based on the data object being common to a predetermined number of client computing devices; and,
storing the modified number of instances of the data object in the one or more secondary storage devices selected from the plurality of secondary storage devices.
12. The method of claim 7 , further comprising:
storing one or more additional instances of the data object in the one or more secondary storage devices to satisfy the determined number of instances based on the determination that the data is to be aged off; and,
aging the data that is to be aged off, wherein aging comprises moving the data from the first tier of storage to a second tier of storage.
13. A non-transitory, computer-readable medium having computer-executable instructions stored thereon that, when executed by one or more processors, cause a system to perform a plurality of operations comprising:
determining that one or more data objects of a plurality of data objects are subject to a storage policy, wherein the storage policy indicates a number of instances that a particular data object is to be stored in one or more secondary storage devices, wherein the storage policy indicates a number of instances that a particular data object is to be stored in one or more secondary storage devices;
for each data object determined to be subject to the storage policy:
determining a number of instances of each data object to store in one or more secondary storage devices selected from a plurality of secondary storage devices, wherein:
the number of instances of each data object is determined based on the storage policy, and
the number of instances of each data object is greater than or equal to 2;
for each instance of a data object:
determining a secondary storage device in which to store the instance of the data object, wherein determining the secondary storage device in which to store the instance of the data object is based on one or more criterion comprising:
an available storage capacity of the secondary storage device;
a time to store data on the secondary storage device;
a throughput of storage operations performed on the secondary storage device; and,
a time to access stored on the secondary storage device;
storing the instance of the data object in the determined secondary storage device;
receiving a request to age the data object, wherein the number of instances of the data object stored in one or more secondary storage devices is equal to or fewer than the determined number of instances, wherein the instances of data object in a first tier of storage; and
aging the data object, w herein the aging comprises moving at least one instance of the data object from the first tier of storage to a second tier of storage, wherein the second tier of storage is associated with an inferior performance compared to the first tier of storage.
14. The non-transitory, computer-readable medium of claim 13 , wherein the plurality of operations further comprises:
generating a unique identifier for the data object;
performing a database lookup of the data object based on the unique identifier to determine a number of instances of the data object that have already been stored; and,
wherein determining the number of instances of each data object to store in the one or more secondary storage devices is further based on the number of instances of the data object that have already been stored.
15. The non-transitory, computer-readable medium of claim 13 , wherein each instance of the data object is stored in a different secondary storage device selected from the plurality of secondary storage devices.
16. The non-transitory, computer-readable medium of claim 13 , wherein determining the secondary storage device in which to store the instance of the data object is based on the storage policy to which the data object is subject.
17. The non-transitory, computer-readable medium of claim 13 , wherein the plurality of operations further comprises:
modifying the determined number of instances of a data object selected from the plurality of data objects based on the data object being common to a predetermined number of client computing devices; and,
storing the modified number of instances of the data object in the one or more secondary storage devices selected from the plurality of secondary storage devices.