Scaling out data protection infrastructure
Embodiments for optimizing data storage instances in a cloud environment in which metadata is stored and accessed separately from content data in multiple different instances of data storage units. A metadata and content data storage instance optimization process determines the status of different instances of virtual storage resources for both metadata and content data. Full instances are powered down when they are not needed, empty instances are deleted, and data of partially full instances is moved to other appropriate instances to create empty instances that can be deleted. The data storage instance optimization process is provided as part of a data protection search process that provides an execution environment and user interface to the optimization process.
1. A computer-implemented method of optimizing data storage instances in a cloud-computing network, comprising:
storing, in a first virtual datastore, metadata of data to be protected by an end-to-end data protection system;
storing, in a second virtual datastore, content data corresponding to respective stored metadata;
utilizing a data protection search function to access data based on metadata indexes and content data indexes for the stored metadata and stored content data;
maintaining a plurality of different storage instances for the data access, wherein each instance has a respective associated cost and resource requirement;
periodically checking, by a storage instance optimization process, the plurality of storage instances containing the metadata and content data, wherein the metadata is stored in on-premise storage and the content data is stored in cloud storage;
scaling the storage instances by adding metadata storage and data storage in accordance with memory usage by the metadata and content data, wherein the cloud storage is charged to a subscribing user based on usage;
identifying full storage instances of the plurality of storage instances;
powering off disk drives of the identified full storage instances for use used by direct access operations or background backup operations;
powering on the powered off disk drives of the full storage instance upon usage of data stored on the powered off disk drives;
identifying non-full storage instances of the plurality of storage instances;
moving data from a non-full storage instance to a fuller storage instance with sufficient spare storage capacity to create an empty storage instance; and
deleting the empty storage instance.
2. The method of claim 1 wherein the end-to-end data protection system comprises backup server executing a deduplication backup system.
3. The method of claim 1 wherein the first virtual datastore and second virtual datastore are each implemented by one or more virtual machines in the network.
4. The method of claim 3 wherein the step of periodically checking is performed by a monitoring component, and wherein a defined period in on the order of daily or weekly checks.
5. The method of claim 4 wherein the defined period conforms to a garbage collection process of the deduplication backup system.
6. The method of claim 1 wherein the metadata stores attributes of the file selected from the group consisting of: file name and directory path to uniquely identify a file stored in an associated instance.
7. The method of claim 1 wherein the data protection search function provides an operating environment and user interface for the computer-implemented method.
8. The method of claim 1 wherein the plurality of storage instances comprises a large-capacity storage cluster accessed by the search function executing data access requests by a user through the user interface.
9. The method of claim 8 wherein the search function utilizes indexes created for the metadata and/or or the content data in their respective storage devices.
10. A system for optimizing data storage instances in a cloud-computing network, comprising:
a first virtual datastore storing metadata of data to be protected by an end-to-end data protection system;
a second virtual datastore storing content data corresponding to respective stored metadata;
a data protection search function accessing data based on metadata indexes and content data indexes for the stored metadata and stored content data;
a plurality of storage instances maintained for the data access, wherein each instance has a respective associated cost and resource requirement;
a monitor periodically checking the plurality of storage instances containing the metadata and content data storage devices, wherein the metadata is stored in on-premise storage and the content data is stored in cloud storage, the system scaling the storage instances by adding metadata storage and data storage in accordance with memory usage by the metadata and content data, wherein the cloud storage is charged to a subscribing user based on usage, and the monitor further identifying full storage instances of the plurality of storage instances, and non-full storage instances of the plurality of storage instances;
a power component powering off disk drives the identified full storage instances, and powering on the powered-off disk drives of the full storage instance when accessed for use used by direct access operations or background backup operations;
an interface moving data from a non-full storage instance to a fuller storage instance with sufficient spare storage capacity to create an empty storage instance; and
a storage instance optimization component deleting the empty storage instance.
11. The system of claim 10 wherein the end-to-end data protection system comprises backup server executing a deduplication backup system, and wherein the first virtual datastore and second virtual datastore are each implemented by one or more virtual machines in the network.
12. The system of claim 10 wherein the monitor periodically checks according to a defined period in on the order of daily or weekly checks.
13. The system of claim 12 wherein the defined period conforms to a garbage collection process of the deduplication backup system.
14. The system of claim 10 wherein the metadata stores attributes of the file selected from the group consisting of: file name and directory path to uniquely identify a file stored in an associated instance.
15. The system of claim 10 further wherein the data protection search component provides an operating environment and user interface optimization component.
16. The system of claim 10 wherein the plurality of storage instances comprises a large-capacity storage cluster accessed by the search function facilitating executing data access requests by a user through the user interface.
17. The system of claim 16 wherein the search component utilizes indexes created for the metadata and/or or the content data in their respective storage devices.
18. A computer program product, comprising a non-transitory computer-readable medium having a computer-readable program code embodied therein, the computer-readable program code adapted to be executed by one or more processors to
storing, in a first virtual datastore, metadata of data to be protected by an end-to-end data protection system;
storing, in a second virtual datastore, content data corresponding to respective stored metadata;
utilizing a data protection search function to access data based on metadata indexes and content data indexes for the stored metadata and stored content data;
maintaining a plurality of different storage instances for the data access, wherein each instance has a respective associated cost and resource requirement;
periodically checking, by a storage instance optimization process, the plurality of storage instances containing the metadata and content data, wherein the metadata is stored in on-premise storage and the content data is stored in cloud storage;
scaling the storage instances by adding metadata storage and data storage in accordance with memory usage by the metadata and content data, wherein the cloud storage is charged to a subscribing user based on usage;
identifying full storage instances of the plurality of storage instances;
powering off disk drives of the identified full storage instances for use used by direct access operations or background backup operations;
powering on the powered off disk drives of the full storage instance upon usage of data stored on the powered off disk drives;
identifying non-full storage instances of the plurality of storage instances;
moving data from a non-full storage instance to a fuller storage instance with sufficient spare storage capacity to create an empty storage instance; and
deleting the empty storage instance.