Distributed framework for task splitting and task assignments in a content indexing system
An improved content indexing (CI) system is disclosed herein. For example, the improved CI system may include a distributed architecture of client computing devices, media agents, a single backup and CI database, and a pool of servers. After a file backup occurs, the backup and CI database may include file metadata indices and other information associated with backed up files. Servers in the pool of servers may, in parallel, query the backup and CI database for a list of files assigned to the respective server that have not been content indexed. The servers may then request a media agent to restore the assigned files from secondary storage and provide the restored files to the servers. The servers may then content index the received restored files. Once the content indexing is complete, the servers can send the content index information to the backup and CI database for storage.
1. An information management system comprising:
one or more non-transitory computer-readable mediums having computer-executable instructions stored thereon; and
one or more hardware processors that, having executed the computer-executable instructions, cause the system to perform a plurality of operations comprising:
determining a controller content indexing proxy that is available to perform content indexing operations;
determining a total number of worker threads executing on the available controller content indexing proxy that are available to perform content indexing operations;
assigning a portion of a total amount data to content index to the available controller content indexing proxy; and
tracking progress of the content indexing performed by the available controller content indexing proxy, wherein the plurality of operations further comprises transmitting a notification indicating the tracked progress, and
wherein the tracked progress comprises one or more of: a percentage of data assigned to the available controller content indexing proxy that has yet to be content indexed, an amount of data assigned to the available controller content indexing proxy that has yet to be content indexed, and a time remaining until the data assigned to the available controller content indexing proxy is content indexed.
2. The system of claim 1 , wherein determining the total number of worker threads is based on at least the total amount of data to content index.
3. The system of claim 1 , further comprising an indexing storage system configured to transmit an indication of the total amount of data to content index to a master content indexing proxy operating in the system.
4. The system of claim 1 , wherein the plurality of operations further comprises:
determining that the available controller content indexing proxy is operating at a performance level below a threshold value based on the tracked progress; and
assigning at least some of the content indexing assigned to the available controller content indexing proxy to another available controller content indexing proxy.
5. The system of claim 1 , wherein the plurality of operations further comprises:
assigning one or more of: a first archive file, a portion of a second archive file, and individual primary data, to the available controller content indexing proxy.
6. A computer-implemented method method comprising:
determining, by a master content indexing proxy, a controller content indexing proxy that is available to perform content indexing operations, wherein the master content indexing proxy comprises one or more hardware processors, and wherein the controller content indexing proxy comprises one or more hardware processors;
determining, by the master content indexing proxy, a total number of worker threads executing on the available controller content indexing proxy that are available to perform content indexing operations;
assigning, by the master content indexing proxy, a portion of a total amount data to content index to the available controller content indexing proxy;
tracking, by the master content indexing proxy, progress of the content indexing performed by the available controller content indexing proxy; and
transmitting a notification indicating the tracked progress, wherein the tracked progress comprises one or more of: a percentage of data assigned to the available controller content indexing proxy that has yet to be content indexed, an amount of data assigned to the available controller content indexing proxy that has yet to be content indexed, and a time remaining until the data assigned to the available controller content indexing proxy is content indexed.
7. The method of claim 6 , wherein determining the total number of worker threads is based on at least the total amount of data to content index.
8. The method of claim 6 , further comprising transmitting, by an indexing storage system, an indication of the total amount of data to content index to the master content indexing proxy.
9. The method of claim 6 , further comprising:
determining that the available controller content indexing proxy is operating at a performance level below a threshold value based on the tracked progress; and
assigning at least some of the content indexing assigned to the available controller content indexing proxy to another available controller content indexing proxy.
10. The method of claim 6 , further comprising:
assigning one or more of: a first archive file, a portion of a second archive file, and individual primary data to the available controller content indexing proxy.
11. An information management system comprising:
one or more non-transitory computer-readable mediums having computer-executable instructions stored thereon; and
one or more hardware processors that, having executed the computer-executable instructions, cause the system to perform a plurality of operations comprising:
determining a controller content indexing proxy that is available to perform content indexing operations;
determining a total number of worker threads executing on the available controller content indexing proxy that are available to perform content indexing operations;
assigning a portion of a total amount data to content index to the available controller content indexing proxy;
tracking progress of the content indexing performed by the available controller content indexing proxy;
determining that the available controller content indexing proxy is operating at a performance level below a threshold value based on the tracked progress; and
assigning at least some of the content indexing assigned to the available controller content indexing proxy to another available controller content indexing proxy.
12. The system of claim 11 , wherein determining the total number of worker threads is based on at least the total amount of data to content index.
13. The system of claim 11 , further comprising an indexing storage system configured to transmit an indication of the total amount of data to content index to a master content indexing proxy operating in the system.
14. The system of claim 13 , wherein the master content indexing proxy comprises one or more hardware processors and is configured to perform one or more of: the determining of the controller content indexing proxy, the determining of the total number of worker threads, the assigning of the portion of the total amount data, the tracking of the progress, the determining that the available controller content indexing proxy is operating below the threshold value, and the assigning to another available controller content indexing proxy.
15. The system of claim 11 , wherein the plurality of operations further comprises: assigning one or more of: a first archive file, a portion of a second archive file, and individual primary data, to the available controller content indexing proxy.
16. The system of claim 11 , wherein the plurality of operations further comprises transmitting a notification indicating the tracked progress.
17. The system of claim 16 , wherein the tracked progress comprises one or more of: a percentage of data assigned to the available controller content indexing proxy that has yet to be content indexed, an amount of data assigned to the available controller content indexing proxy that has yet to be content indexed, and a time remaining until the data assigned to the available controller content indexing proxy is content indexed.
18. The system of claim 1 , further comprising an indexing storage system configured to transmit an indication of the total amount of data to content index to a master content indexing proxy, wherein the master content indexing proxy comprises one or more hardware processors and is configured to perform one or more of: the determining of the controller content indexing proxy, the determining of the total number of worker threads, the assigning of the portion of the total amount data, and the tracking of the progress.