Data analysis control

1. A method for data analysis control, the method comprising:

receiving a predetermined number of search results for a query;

organizing the predetermined number of search results by checksum values of content of the search results in an ascending or a descending order;

using one of a generally final and a generally initial checksum value from the ascending or the descending order of the organized search results to determine a total content number representing an approximate total content of a search related to the query, wherein the total content number is greater than the predetermined number of search results;

using the determined total content number to determine, by a processor, a number of sub-tasks needed to complete a full search of the approximate total content; and

determining a range for each of the sub-tasks and allocating each range to one of a plurality of agents to execute the full search.

2. The method of claim 1 , wherein using one of a generally final and a generally initial checksum value from the ascending or the descending order of the organized search results to determine a total content number representing an approximate total content of a search related to the query further comprises:

using one of a final and an initial checksum value from the ascending or the descending order of the organized search results to determine the total content number representing the approximate total content of the search related to the query.

3. The method of claim 1 , wherein the content of the search includes documents.

4. The method of claim 1 , wherein using the determined total content number to determine a number of sub-tasks needed to complete a full search of the approximate total content further comprises:

determining the number of sub-tasks based on a generally equal distribution of search loads to perform the full search.

5. The method of claim 1 , wherein the determining the range for each of the sub-tasks is based on the determined total content number representing the approximate total content of the search.

6. The method of claim 5 , further comprising:

processing the approximate total content based on the determined range for each of the sub-tasks, wherein the processing is designated a trawl phase and includes partial trawls that correspond to the determined range for each of the sub-tasks; and

marking predetermined statistics for the content during at least one of the partial trawls.

7. The method of claim 6 , wherein the predetermined statistics include duplicate documents in the content.

8. The method of claim 5 , further comprising:

processing the approximate total content based on the determined range for each of the sub-tasks, wherein the processing is designated a trawl phase and includes partial trawls that correspond to the determined range for each of the sub-tasks; and

marking predetermined statistics for the content during a plurality of the partial trawls.

9. The method of claim 8 , further comprising:

combining the marked predetermined statistics in parallel for different partial trawls.

10. The method of claim 5 , further comprising:

processing the approximate total content based on the determined range for each of the sub-tasks, wherein the processing is designated a trawl phase and includes partial trawls that correspond to the determined range for each of the sub-tasks;

combining the partial trawls in a combine phase to generate combined files; and

combining the combined files to generate final files representing final search results for the query.

11. The method of claim 1 , wherein the checksum values are based on a checksum function that provides a generally uniform distribution of the checksum values.

12. A data analysis control apparatus comprising:

a memory storing machine readable instructions to:

receive a predetermined number of search results for a query;

organize the predetermined number of search results by checksum values of content of the search results in an ascending or a descending order;

use one of a generally final and a generally initial checksum value from the ascending or the descending order of the organized search results to determine a total content number representing an approximate total content of a search related to the query, wherein the total content number is greater than the predetermined number of search results;

use the determined total content number to determine a number of sub-tasks needed to complete a full search of the approximate total content by determining the number of sub-tasks based on a generally equal distribution of search loads to perform the full search; and

determine a range for each of the sub-tasks and allocate each of the ranges to one of a plurality of agents to execute the full search; and

a processor to implement the machine readable instructions.

13. The data analysis control apparatus according to claim 12 , wherein the determining the range for each of the sub-tasks is based on the determined total content number representing the approximate total content of the search.

14. The data analysis control apparatus according to claim 12 , wherein the checksum values are based on a checksum function that provides a generally uniform distribution of the checksum values.

15. A non-transitory computer readable medium having stored thereon machine readable instructions to provide data analysis control, the machine readable instructions, when executed, cause a computer system to:

receive a predetermined number of search results for a query;

organize the predetermined number of search results by checksum values of content of the search results in an ascending or a descending order;

use one of a generally final and a generally initial checksum value from the ascending or the descending order of the organized search results to determine a total content number representing an approximate total content of a search related to the query, wherein the total content number is greater than the predetermined number of search results;

use the determined total content number to determine, by a processor, a number of sub-tasks needed to complete a full search of the approximate total content; and

determine a range for each of the sub-tasks based on the determined total content number representing the approximate total content of the search and allocate each range to one of a plurality of agents to execute the full search.