Systems and methods for load-balancing by secondary processors in parallel document indexing

1. A data processing system for parallelizing document indexing, comprising:

a primary processor for receiving a list of data, the data having embedded data associated therewith, and creating a work item based upon the list of data;

a first secondary processor to process the work item as provided by the primary processor, the first secondary processor further comprising a data processor to:

extract the embedded data;

determine a characteristic of the embedded data;

determine that the embedded data should be processed by a second secondary processor;

provide a status message to the primary processor causing the primary processor to create a new work item;

a second secondary processor configured to receive the new work item and process the embedded data; and

a messaging module to exchange the status message between the primary processor and the first secondary processor.

2. The data processing system of claim 1 , wherein the primary processor is further comprised of a job queue, a work item queue, and a message processing module.

3. The data processing system of claim 2 , wherein the job queue and the work item queue store the list of data.

4. The data processing system of claim 1 , wherein the data comprises an electronic file embedded within another file.

5. The data processing system of claim 1 , wherein the messaging module is further configured to exchange a second status message comprising one selected from the list consisting of: a processed work item, a processed work item related to the embedded data, a result of the processed work items, a request for the work item, a new work item created by the primary processor, and a notification relating to the completion of processed work items.

6. The data processing system of claim 1 , wherein the work item is comprised of the physical location of the data stored in a data storage medium, at least one data identifier to distinguish the location of the embedded data, and an identifier associated with the data.

7. The data processing system of claim 1 , wherein the first secondary processor initiates the messaging module with the primary processor.

8. The data processing system of claim 2 , wherein the messaging module appends the job queue or the work item queue based upon the status message.

9. The data processing system of claim 1 , wherein the messaging module aggregates results provided by the first secondary processor and the second secondary processor.

10. The data processing system of claim 1 , further comprising a failure tolerance mechanism.

11. The data processing system of claim 1 , wherein each of the first secondary processor and the another secondary processor is configured to ask the primary processor for a new work item when it is idle.

12. A method for parallelizing document indexing in a data processing system, comprising:

receiving a list of data having embedded data associated therewith;

creating, using a primary processor, a work item based upon the list of data, the work item including the physical location of the data stored in a data storage medium, at least one data identifier to distinguish the location of the embedded data, and an identifier associated with the data;

exchanging from the primary processor to a secondary processor a status message comprising a request to process the work item;

using the secondary processor to process the work item, determine a characteristic of the embedded data, and determine that the embedded data should be processed by another secondary processor;

exchanging from the secondary processor to the primary processor a status message causing the primary processor to create a new work item;

using a second secondary processor to receive the new work item and process the embedded data.

13. The method of claim 12 , further comprising initializing a job queue and a work item queue to store the list of data.

14. The method of claim 12 , further comprising:

using the secondary processor to determine that the work item should be processed by another secondary processor;

exchanging from the secondary processor to the primary processor a status message indicating that the work item should be processed by another secondary processor, the status message comprising the work item; and

using the second secondary processor to receive the work item from the primary processor and process the data item including the embedded data.

15. The method of claim 12 , further comprising providing a failure tolerance mechanism.

16. The method of claim 12 , wherein the data comprises an exchangeable database file.

17. The method of claim 12 , wherein the data comprises a disk image file.

18. The method of claim 12 , wherein the associated data comprises at least one electronic file.

19. A parallel document indexing load-balancing system comprising:

a primary processor configured to receive and store a list of data and create a work item that specifies a data item to be processed, the data item comprising embedded data;

a secondary processor configured to

receive the work item and extract the embedded data,

determine, based on a characteristic of the embedded data, that the work item should be processed by another secondary processor, and

provide a status message to the primary processor indicating that the work item should be processed by another secondary processor, the status message comprising the work item; and

a second secondary processor configured to receive the work item from the primary processor and process the data item including the embedded data.

20. The data processing system of claim 19 , wherein the embedded data item comprises one selected from the list consisting of a: an attachment in an email file; partitions within a disk image; an email message inside of a PST file; and documents within a directory.

21. The system of claim 20 , wherein the secondary processor classifies the embedded data for processing as INLINE, WORK 13 ITEM, or PROCESSING 13 JOB based on a complexity of the embedded data.

22. The system of claim 19 , wherein each of the secondary processors is configured to ask the primary processor for a new work item when it is idle.

23. The method of claim 12 , wherein each of the secondary processors is configured to ask the primary processor for a new work item when it is idle.