Distributed framework for data proximity-based task splitting in a content indexing system
An improved content indexing (CI) system is disclosed herein. For example, the improved CI system may include a distributed architecture of client computing devices, media agents, a single backup and CI database, and a pool of servers. After a file backup occurs, the backup and CI database may include file metadata indices and other information associated with backed up files. Servers in the pool of servers may, in parallel, query the backup and CI database for a list of files assigned to the respective server that have not been content indexed. The servers may then request a media agent to restore the assigned files from secondary storage and provide the restored files to the servers. The servers may then content index the received restored files. Once the content indexing is complete, the servers can send the content index information to the backup and CI database for storage.
1. A networked information management system for content indexing data, the networked information management system comprising:
a master content indexing proxy having one or more first hardware processors, wherein the master content indexing proxy is configured to:
identify, from a plurality of controller content indexing proxies, one or more controller content indexing proxies that are available to perform content indexing operations,
wherein the content indexing operations are executed according to a content indexing policy assigned to client computing device storing primary data,
wherein the content indexing policy indicates which files among the primary data are to be content indexed;
for each available controller content indexing proxy of the one or more controller content indexing proxies,
assign a portion of a total amount of data to content index to the respective available controller content indexing proxy based on at least one of:
the total amount of data to content index, and
a total number of available controller content indexing proxies, and
transmit an instruction to the respective available controller content indexing proxy indicating the portion of the total amount of data to content index assigned to the respective available controller content indexing proxy; and
the one or more controller content indexing proxies, wherein the one or more controller content indexing proxies is configured to:
cause a restoration of secondary copies corresponding to the assigned portion of the total amount of data to content index to the respective available controller content indexing proxy; and
cause content indexing to performed on the restored secondary copies.
2. The networked information management system of claim 1 , wherein the master content indexing proxy is further configured to: determine a total number of worker threads executing on the respective available controller content indexing proxy that are available to perform content indexing operations.
3. The networked information management system of claim 2 , wherein master content indexing proxy is further configured to: assign a portion of the total amount of data to content index to the respective available controller content indexing proxy based on the total number of worker threads executing on the respective available controller content indexing proxy that are available to perform content indexing operations.
4. The networked information management system of claim 1 , wherein the total amount of data to content index comprises at least one of a total number of archive files that include secondary copies that correspond with primary data to be content indexed or a number of secondary copies that are associated with each archive file that correspond with primary data to be content indexed.
5. The networked information management system of claim 1 , wherein the networked information management system further comprises:
an indexing storage system in communication with the master content indexing proxy,
wherein the indexing storage system is configured to cause the indexing storage system to transmit an indication of the total amount of data to content index to the master content indexing proxy.
6. The networked information management system of claim 1 , wherein the master content indexing proxy is further configured to track progress of content indexing performed by the one or more controller content indexing proxies.
7. The networked information management system of claim 6 , wherein the master content indexing proxy is further configured to:
determine that a first available controller content indexing proxy is operating at a performance level below a threshold value based on the tracked progress; and
assign at least some of content indexing tasks assigned to the first available controller content indexing proxy to another available controller content indexing proxy.
8. The networked information management system of claim 7 , wherein the tracked progress comprises at least one of:
a percentage of the portion of data assigned to the controller content indexing proxy that has yet to be content indexed;
an amount of the portion of data assigned to the controller content indexing proxy that has yet to be content indexed; and,
a time remaining until the portion of data assigned to the available controller content indexing proxy is content indexed.
9. The networked information management system of claim 8 , wherein the master content indexing proxy is further configured to transmit a notification indicating the tracked progress.
10. A computer-implemented method for content indexing data, the computer-implemented method comprising:
identifying, from a plurality of controller content indexing proxies, one or more controller content indexing proxies that are available to perform content indexing operations,
wherein the content indexing operations are executed according to a content indexing policy assigned to client computing device storing primary data,
wherein the content indexing policy indicates which files among the primary data are to be content indexed;
for each available controller content indexing proxy of the one or more controller content indexing proxies:
assigning a portion of a total amount of data to content index to the respective available controller content indexing proxy based on at least one of:
the total amount of data to content index, and
a total number of available controller content indexing proxies, and
transmitting an instruction to the respective available controller content indexing proxy indicating the portion of the total amount of data to content index assigned to the respective available controller content indexing proxy;
causing a restoration of secondary copies corresponding to the assigned portion of the total amount of data to content index to the respective available controller content indexing proxy; and
causing content indexing to performed on the restored secondary copies.
11. The computer-implemented method of claim 10 , wherein the computer-implemented method further comprises determining a total number of worker threads executing on the respective available controller content indexing proxy that are available to perform content indexing operations.
12. The computer-implemented method of claim 11 , wherein the computer-implemented method further comprises assigning a portion of the total amount of data to content index to the respective available controller content indexing proxy based on the total number of worker threads executing on the respective available controller content indexing proxy that are available to perform content indexing operations.
13. The computer-implemented method of claim 10 , wherein the total amount of data to content index comprises at least one of a total number of archive files that include secondary copies that correspond with primary data to be content indexed or a number of secondary copies that are associated with each archive file that correspond with primary data to be content indexed.
14. The computer-implemented method of claim 10 , wherein the computer-implemented method further comprises: causing an indexing storage system to transmit an indication of the total amount of data to content index to a master content indexing proxy.
15. The computer-implemented method of claim 10 , wherein computer-implemented method further comprises tracking progress of content indexing performed by the one or more controller content indexing proxies.
16. The computer-implemented method of claim 15 , computer-implemented method further comprises:
determining that a first available controller content indexing proxy is operating at a performance level below a threshold value based on the tracked progress; and
assigning at least some of content indexing tasks assigned to the first available controller content indexing proxy to another available controller content indexing proxy.
17. The computer-implemented method of claim 16 , wherein the tracked progress comprises at least one of:
a percentage of the portion of data assigned to the controller content indexing proxy that has yet to be content indexed;
an amount of the portion of data assigned to the controller content indexing proxy that has yet to be content indexed; and,
a time remaining until the portion of data assigned to the available controller content indexing proxy is content indexed.
18. The computer-implemented method of claim 17 , wherein the computer-implemented method further comprises transmitting a notification indicating the tracked progress.