Distributed architecture for tracking content indexing
An improved content indexing system is disclosed herein that content indexing system combines the functionality of the backup metadata database and the content index database into a single backup and content index database to avoid the need to perform synchronization operations. By using a single backup and content index database, the content indexing system also reduces the computing performance costs that would be associated with the synchronization operations as the amount of indexed content increases, thereby solving scalability issues.
1. A networked information management system for tracking content indexing, the networked information management system comprising:
an indexing storage system having one or more first hardware processors, wherein the indexing storage system is configured with first computer-executable instructions that, when executed, cause the indexing storage system to:
receive an indication that a first file has been backed up by a first computing device that executes a media agent;
add a first entry in a backup and content indexing database corresponding to the first file that has been backed up subsequent to the first file being backed up, wherein the first entry comprises an indication that the corresponding first file has not been content indexed, and wherein the backup and content indexing database comprises a plurality of other entries;
receive a request for a total amount of data to content index;
determine that the first entry in the backup and content indexing database comprises the indication that the corresponding first file has not been content indexed;
determine that a second entry in the plurality of other entries comprises an indication that a corresponding second file has not been content indexed; and
transmit a response to the request providing the total amount of data to content index, wherein the total amount of data to content index is determined based at least in part on the first file and the second file; and
a master content indexing proxy in communication with the indexing storage system, wherein the master content indexing proxy has one or more second hardware processors, wherein the master content indexing proxy is configured with second computer-executable instructions that, when executed, cause the master content indexing proxy to transmit the request for the total amount of data to content index.
2. The networked information management system of claim 1 , wherein the first computer-executable instructions, when executed, further cause the indexing storage system to determine that the first file and the second file correspond to criteria included in a context indexing policy.
3. The networked information management system of claim 1 , wherein the total amount of data to context index comprises at least one of a total number of archive files that include secondary copies that correspond with primary data to be context indexed or a number of secondary copies that are associated with each archive file that correspond with primary data to be context indexed.
4. The networked information management system of claim 1 , wherein the first computer-executable instructions, when executed, further cause the indexing storage system to receive at least one of file metadata associated with the first file or secondary copy metadata associated with the first file and generated by the first computing device.
5. The networked information management system of claim 4 , wherein the first computer-executable instructions, when executed, further cause the indexing storage system to store at least one of the file metadata or the secondary copy metadata in the first entry in the backup and content indexing database.
6. The networked information management system of claim 1 , wherein the second computer-executable instructions, when executed, further cause the master content indexing proxy to identify at least one of a total number of controller content indexing proxies available to perform content indexing tasks or a total number of worker threads executing on each controller content indexing proxy available to perform content indexing tasks.
7. The networked information management system of claim 6 , wherein the second computer-executable instructions, when executed, further cause the master content indexing proxy to split the total amount of data to content index for assignment to different controller content indexing proxies available to perform content indexing tasks.
8. The networked information management system of claim 7 , wherein the second computer-executable instructions, when executed, further cause the master content indexing proxy to assign the first file to a first controller content indexing proxy available to perform content indexing tasks and assign the second file to a second controller content indexing proxy available to perform content indexing tasks.
9. The networked information management system of claim 8 , wherein the second computer-executable instructions, when executed, further cause the master content indexing proxy to track and report on progress of content indexing performed by the first controller content indexing proxy and by the second controller content indexing proxy.
10. The networked information management system of claim 1 , wherein the first entry comprises a status flag that indicates that the first file has not been content indexed.
11. A computer-implemented method for tracking content indexing, the computer-implemented method comprising:
receiving an indication that a first file has been backed up by a first computing device that executes a media agent;
adding a first entry in a backup and content indexing database corresponding to the first file that has been backed up subsequent to the first file being backed up, wherein the first entry comprises an indication that the corresponding first file has not been content indexed, and wherein the backup and content indexing database comprises a plurality of other entries;
receiving a request for a total amount of data to content index;
determining that the first entry in the backup and content indexing database comprises the indication that the corresponding first file has not been content indexed;
determining that a second entry in the plurality of other entries comprises an indication that a corresponding second file has not been content indexed; and
transmitting a response to the request providing the total amount of data to content index, wherein the total amount of data to content index is determined based at least in part on the first file and the second file.
12. The computer-implemented method of claim 11 , further comprising determining that the first file and the second file correspond to criteria included in a context indexing policy.
13. The computer-implemented method of claim 11 , wherein the total amount of data to context index comprises at least one of a total number of archive files that include secondary copies that correspond with primary data to be context indexed or a number of secondary copies that are associated with each archive file that correspond with primary data to be context indexed.
14. The computer-implemented method of claim 11 , wherein receiving an indication that a first file has been backed up by a first computing device that executes a media agent further comprises receiving at least one of file metadata associated with the first file or secondary copy metadata associated with the first file and generated by the first computing device.
15. The computer-implemented method of claim 14 , further comprising storing at least one of the file metadata or the secondary copy metadata in the first entry in the backup and content indexing database.
16. The computer-implemented method of claim 11 , further comprising identifying at least one of a total number of controller content indexing proxies available to perform content indexing tasks or a total number of worker threads executing on each controller content indexing proxy available to perform content indexing tasks.
17. The computer-implemented method of claim 16 , further comprising splitting the total amount of data to content index for assignment to different controller content indexing proxies available to perform content indexing tasks.
18. The computer-implemented method of claim 17 , further comprising:
assigning the first file to a first controller content indexing proxy available to perform content indexing tasks;
assigning the second file to a second controller content indexing proxy available to perform content indexing tasks; and
tracking and reporting on progress of content indexing performed by the first controller content indexing proxy and by the second controller content indexing proxy.
19. The computer-implemented method of claim 18 , wherein the first controller content indexing proxy causes content indexing to be performed on a restored secondary copy of the first file in an independent format.
20. The computer-implemented method of claim 11 , wherein receiving a request for a total amount of data to content index further comprises receiving a request for a total amount of data in a first mailbox to content index.