Heterogeneous indexing and load balancing of backup and indexing resources
Indexing preferences generally associate each data source with a type of indexing technology and/or with an index/catalog and/or with a computing device that hosts the index/catalog for tracking backup data generated from the source data. Indexing preferences govern which index/catalog receives transaction logs for a given storage operation. Thus, indexing destinations are defined granularly and flexibly in reference to the source data. Load balancing without user intervention assures that the various index/catalogs are fairly distributed in the illustrative backup systems by autonomously initiating migration jobs. Criteria for initiating migration jobs are based on past usage and going-forward trends. An illustrative migration job re-associates data sources with a different destination media agent and/or index/catalog, including transferring some or all relevant transaction logs and/or indexing information from the old host to the new host.
1. A computer-implemented method comprising:
by a first computing device in a storage system, wherein the first computing device comprises one or more hardware processors:
generating first backup data from a first data source;
storing the first backup data to one or more storage devices that are communicatively coupled to the first computing device;
generating one or more first log files, wherein the one or more first log files are based on one or more of: the generating of the first backup data and the storing of the first backup data;
accessing indexing preferences configured in the storage system, wherein the indexing preferences indicate a first index for a first type of data source and further indicate a second index for a second type of data source, which differs from the first type:
based on the indexing preferences and further based on determining that the first data source is of the first type, transmitting the one or more first log files to the first index, wherein the first index is updated based on information in the one or more first log files, and wherein the first index is based on a first type of indexing technology, which is associated with the first type of data source;
generating second backup data from a second data source of the second type, and further generating one or more second log files based on one or more of: the generating of the second backup data and storing of the second backup data;
based on the indexing preferences and further based on determining that the second data source is of the second type of data source, transmitting the one or more second log files to the second index, wherein the second index is based on a second type of indexing technology that differs from the first type of indexing technology, wherein the second type of indexing technology is associated with the second type of data source, and wherein the second index is updated based on information in the one or more second log files;
responsive to a request to restore a first backup file from the first backup data, accessing the first index to identify portions of the first backup data that correspond to the first backup file; and
restoring the first backup file from the portions of the first backup data identified in the first index.
2. The computer-implemented method of claim 1 , wherein the storage system comprises a plurality of types of heterogeneous indexing technologies that includes the first type of indexing technology and the second type of indexing technology; and further comprising: for a given data source being backed up, choosing, by the storage system, from the plurality of types of heterogeneous indexing technologies based on a type of the given data source.
3. The computer-implemented method of claim 1 ,
wherein the first type of indexing technology is a ctree index and, in the indexing preferences, is associated with network attached storage; and
wherein the second type of indexing technology is a solr index and, in the indexing preferences, is associated with one or more web-based file storage and synchronization services.
4. The computer-implemented method of claim 1 ,
wherein the first type of indexing technology is a ctree index and, in the indexing preferences, is associated with network attached storage; and
wherein the second type of indexing technology comprises a database management system based on Structured Query Language (SQL) technology and, in the indexing preferences, is associated with one or more structured databases.
5. The computer-implemented method of claim 1 , wherein the first computing device transmits the one or more first log files to the first index and transmits the one or more second log files to the second index based on a choice of indexing technology, wherein the choice of indexing technology is based on a type of data of each data source from which backup data is generated in the storage system.
6. The computer-implemented method of claim 1 , wherein a storage manager instructs the first computing device to transmit the one or more first log files to the first index and to transmit the one or more second log files to the second index based on the indexing preferences in the storage system that govern a choice of index, wherein the choice of index is based on a type of each data source from which backup data is generated in the storage system, and wherein the storage manager comprises one or more hardware processors.
7. The computer-implemented method of claim 1 , wherein the first computing device is configured to choose between the first index and the second index based on a type of data source being backed up.
8. The computer-implemented method of claim 1 , wherein the storage system is configured with heterogeneous indexing technologies, including the first type of indexing technology and the second type of indexing technology.
9. The computer-implemented method of claim 8 , wherein the first computing device is configured to choose between the first index and the second index based on a type of data source being backed up and further based on a type of indexing technology of the first index and the second index, respectively.
10. The computer-implemented method of claim 8 , wherein one of: the first index and the second index is chosen for a given data source being backed up, based on a type of the given data source being backed up and further based on a type of indexing technology of the first index and the second index, respectively.
11. A storage system comprising:
a first computing device, wherein the first computing device comprises one or more hardware processors, and wherein the first computing device is configured to:
generate first backup data from first primary data;
store the first backup data to one or more storage devices;
generate one or more first log files, wherein the one or more first log files are based on one or more of: generating of the first backup data and storing of the first backup data to the one or more storage devices;
accessing indexing preferences configured in the storage system, wherein the indexing preferences indicate a first index for a first type of data and further indicate a second index for a second type of data, which differs from the first type;
based on the indexing preferences and further based on determining that the first primary data is of the first type,
transmit the one or more first log files to the first index, wherein the first index is updated based on information in the one or more first log files, and wherein the first index is based on a first type of indexing technology, which is associated with the first type of data;
generate second backup data from second primary data of the second type of data, and further generate one or more second log files based on one or more of: generating of the second backup data and storing of the second backup data;
based on the indexing preferences and further based on determining that the second primary data is of the second type of data, transmit the one or more second log files to the second index, wherein the second index is based on a second type of indexing technology that differs from the first type of indexing technology, wherein the second type of indexing technology is associated with the second type of data, and wherein the second index is updated based on information in the one or more second log files; and
responsive to a request to restore first data from the first backup data, access the first index to locate portions of the first backup data that correspond to the first data; and
restore the first data from the portions of the first backup data located by access to the first index.
12. The storage system of claim 11 , wherein the storage system further comprises: a plurality of types of heterogeneous indexing technologies that includes the first type of indexing technology and the second type of indexing technology; and wherein for a given primary data being backed up, the storage system is configured to choose from the plurality of types of heterogeneous indexing technologies based on a type of data of the given primary data.
13. The storage system of claim 11 ,
wherein the first type of indexing technology is a ctree index and, in the indexing preferences, is associated with primary data that is stored in network attached storage; and
wherein the second type of indexing technology is a solr index and, in the indexing preferences, is associated with primary data that is stored in one or more web-based file storage and synchronization services.
14. The storage system of claim 11 ,
wherein the first type of indexing technology is a ctree index and, in the indexing preferences, is associated with primary data that is stored in network attached storage; and
wherein the second type of indexing technology comprises a database management system based on Structured Query Language (SQL) technology and, in the indexing preferences, is associated with primary data in one or more structured databases.
15. The storage system of claim 11 , wherein the first computing device transmits the one or more first log files to the first index and transmits the one or more second log files to the second index based on a choice of indexing technology, wherein the choice of indexing technology is based on a type of primary data from which backup data is generated in the storage system.
16. The storage system of claim 11 , further comprising: a storage manager that comprises one or more hardware processors; and wherein the storage manager is configured to:
instruct the first computing device to transmit the one or more first log files to the first index and to transmit the one or more second log files to the second index based on the indexing preferences in the storage system that govern a choice of index, wherein the choice of index is based on a type of primary data from which backup data is generated in the storage system.
17. The storage system of claim 11 , wherein the first computing device is configured to choose between the first index and the second index based on a type of primary data being backed up.
18. The storage system of claim 11 , wherein the storage system is configured with heterogeneous indexing technologies, including the first type of indexing technology and the second type of indexing technology.
19. The storage system of claim 18 , wherein the first computing device is configured to choose between the first index and the second index based on a type of primary data being backed up and further based on a type of indexing technology of the first index and the second index, respectively.
20. The storage system of claim 18 , wherein one of: the first index and the second index is chosen for a given primary data being backed up, based on a type of the given primary data and further based on a type of indexing technology of the first index and the second index, respectively.