Distributed framework for data proximity-based task splitting in a content indexing system
An improved content indexing (CI) system is disclosed herein. For example, the improved CI system may include a distributed architecture of client computing devices, media agents, a single backup and CI database, and a pool of servers. After a file backup occurs, the backup and CI database may include file metadata indices and other information associated with backed up files. Servers in the pool of servers may, in parallel, query the backup and CI database for a list of files assigned to the respective server that have not been content indexed. The servers may then request a media agent to restore the assigned files from secondary storage and provide the restored files to the servers. The servers may then content index the received restored files. Once the content indexing is complete, the servers can send the content index information to the backup and CI database for storage.
1. A networked information management system for content indexing data, the networked information management system comprising:
one or more non-transitory, computer-readable mediums having computer-executable instructions stored thereon; and,
one or more processors that, having executed the computer-executable instructions, configure the networked information management system to perform a plurality of operations comprising:
receiving an indication of an amount of data to content index;
determining a controller content indexing proxy is available to perform content indexing operations, wherein:
the controller content indexing proxy is executed by a first computing device that executes a media agent, and
the media agent manages at least a subset of the amount of data to content index;
assigning the subset of the amount of data to content index to the controller content indexing proxy such that the media agent restores secondary copies corresponding to the subset of the amount of data and provides the restored secondary copies to the controller content indexing proxy; and
transmitting an instruction to the controller content indexing proxy indicating that the subset of the amount of data to content index is assigned to the controller content indexing proxy.
2. The networked information management system of claim 1 , wherein the plurality of operations further comprises:
transmitting, by an indexing storage system, the indication of the amount of data to content index to a master content indexing proxy, wherein the master content indexing proxy determines whether the controller content indexing proxy is available to perform the content indexing operations.
3. The networked information management system of claim 1 , wherein the plurality of operations further comprises:
tracking progress of content indexing performed by the controller content indexing proxy.
4. The networked information management system of claim 3 , wherein the plurality of operations further comprises:
transmitting a notification indicating the tracked progress.
5. The networked information management system of claim 3 , wherein the tracked progress comprises one of:
a percentage of the subset of the amount of data assigned to the controller content indexing proxy that has yet to be content indexed;
an amount of the subset of the amount of data assigned to the controller content indexing proxy that has yet to be content indexed; or,
a time remaining until the subset of the amount of data assigned to the available controller content indexing proxy is content indexed.
6. The networked information management system of claim 3 , wherein the plurality of operations further comprises:
determining that the controller content indexing proxy is operating at a performance level below a threshold value based on the tracked progress; and
assigning at least some of the subset of the amount of data assigned to the controller content indexing proxy to another controller content indexing proxy.
7. The networked information management system of claim 1 , wherein the plurality of operations further comprises:
assigning one of a first archive file, a portion of a second archive file, or individual primary data to the controller content indexing proxy.
8. A computer-implemented method for content indexing data, the method comprising:
receiving an indication of an amount of data to content index;
determining a controller content indexing proxy is available to perform content indexing operations, wherein:
the controller content indexing proxy is executed by a first computing device that executes a media agent, and
the media agent manages at least a subset of the amount of data to content index;
assigning the subset of the amount of data to content index to the controller content indexing proxy such that the media agent restores secondary copies corresponding to the subset of the amount of data and provides the restored secondary copies to the controller content indexing proxy; and
transmitting an instruction to the controller content indexing proxy indicating that the subset of the amount of data to content index is assigned to the controller content indexing proxy.
9. The computer-implemented method of claim 8 , further comprising:
transmitting, by an indexing storage system, the indication of the amount of data to content index to a master content indexing proxy, wherein the master content indexing proxy determines whether the controller content indexing proxy is available to perform the content indexing operations.
10. The computer-implemented method of claim 8 , further comprising:
tracking progress of content indexing performed by the controller content indexing proxy.
11. The computer-implemented method of claim 10 , further comprising:
transmitting a notification indicating the tracked progress.
12. The computer-implemented method of claim 10 , wherein the tracked progress comprises one of:
a percentage of the subset of the amount of data assigned to the controller content indexing proxy that has yet to be content indexed;
an amount of the subset of the amount of data assigned to the controller content indexing proxy that has yet to be content indexed; or,
a time remaining until the subset of the amount of data assigned to the available controller content indexing proxy is content indexed.
13. The computer-implemented method of claim 10 , further comprising:
determining that the controller content indexing proxy is operating at a performance level below a threshold value based on the tracked progress; and
assigning at least some of the subset of the amount of data assigned to the controller content indexing proxy to another controller content indexing proxy.
14. The computer-implemented method of claim 8 , further comprising:
assigning one of a first archive file, a portion of a second archive file, or individual primary data to the controller content indexing proxy.
15. A computer-readable medium having computer-executable instructions stored thereon that, when executed by one or more processors, cause a networked information management system to perform a plurality of operations comprising:
receiving an indication of an amount of data to content index;
determining a controller content indexing proxy is available to perform content indexing operations, wherein:
the controller content indexing proxy is executed by a first computing device that executes a media agent, and
the media agent manages at least a subset of the amount of data to content index;
assigning the subset of the amount of data to content index to the controller content indexing proxy such that the media agent restores secondary copies corresponding to the subset of the amount of data and provides the restored secondary copies to the controller content indexing proxy; and
transmitting an instruction to the controller content indexing proxy indicating that the subset of the amount of data to content index is assigned to the controller content indexing proxy.
16. The computer-readable medium of claim 15 , wherein the plurality of operations further comprises:
transmitting, by an indexing storage system, the indication of the amount of data to content index to a master content indexing proxy, wherein the master content indexing proxy determines whether the controller content indexing proxy is available to perform the content indexing operations.
17. The computer-readable medium of claim 15 , wherein the plurality of operations further comprises:
tracking progress of content indexing performed by the controller content indexing proxy.
18. The computer-readable medium of claim 17 , wherein the plurality of operations further comprises:
transmitting a notification indicating the tracked progress.
19. The computer-readable medium of claim 17 , wherein the tracked progress comprises one of:
a percentage of the subset of the amount of data assigned to the controller content indexing proxy that has yet to be content indexed;
an amount of the subset of the amount of data assigned to the controller content indexing proxy that has yet to be content indexed; or,
a time remaining until the subset of the amount of data assigned to the available controller content indexing proxy is content indexed.
20. The computer-readable medium of claim 17 , wherein the plurality of operations further comprises:
determining that the controller content indexing proxy is operating at a performance level below a threshold value based on the tracked progress; and
assigning at least some of the subset of the amount of data assigned to the controller content indexing proxy to another controller content indexing proxy.