Method, device and computer program product for managing index tables

1. A method, comprising:

facilitating, by a system comprising a processor, receiving respective ones of a first part of documents of an unbounded stream of documents from a document source, wherein the unbounded stream arrives at the system according to a variable arriving rate;

for the respective ones of the first part of documents, generating, by the system, an index request for a first index processing device, wherein the index request causes the first index processing device to index a received document of the first part of documents in a first cached indexing table generated by the first index processing device;

determining, by the system, a generating speed of respective indexing requests for the first part of documents;

based on the variable arriving rate and the generating speed in relation to a first threshold, for a second part of the documents of the unbounded stream, allocating, by the system, a second index processing device to perform indexing of received documents of the second part of documents in a second cached indexing table generated by the second index processing device, wherein indexing the received documents of the second part of the unbounded stream by the second index processing device is performed in parallel with the first index processing device indexing the received document of the first part of documents into the first cached indexing table; and

based on the variable arriving rate and an appropriate number of indexes to achieve an indexing performance requirement for generation of indexes, determining, by the system, whether to merge the first cached indexing table and the second cached indexing table into a first receiving table for searching, or generate a second receiving table and a third receiving table respectively from the first cached indexing table and the second cached indexing table, for searching.

2. The method of claim 1 , wherein the generating speed in relation to the first threshold comprises the generating speed being above the first threshold that was selected to adjust processing resources allocated to the unbounded stream of documents.

3. The method of claim 1 , further comprising:

determining, by the system, a surplus part of the first cached indexing table, wherein an amount of data corresponding to the surplus part is greater than a second threshold amount of data;

merging, by the system, first entries in the surplus part into second entries in the first receiving table;

associating, by the system, the first receiving table and documents associated with the surplus part; and

deleting, by the system, the surplus part.

4. The method of claim 1 , further comprising:

in response to determining that an amount of data corresponding to the first cached indexing table and the second cached indexing table is greater than a third threshold amount of data, ceasing, by the system, adding a new entry into the first cached indexing table and the second cached indexing table; and

generating, by the system, a new indexing table at a second index processing device different from the first index processing device.

5. Indexing equipment, comprising:

at least one processing unit; and

at least one memory coupled to the at least one processing unit, that stores executable instructions that, when executed by the at least one processing unit facilitate performance of operations, comprising:

receiving respective ones of a first part of documents of an unbounded stream of documents from a document source, wherein the unbounded stream arrives according to a variable arriving rate,

indexing a received document of the first part of documents in a first cached indexing table,

determining a generating speed of respective indexing requests for the first part of documents;

based on the variable arriving rate and the generating speed in relation to a first threshold, for a second part of the documents of the unbounded stream, indexing received documents of the second part of documents in a second cached indexing table, wherein indexing the received documents of the second part of the unbounded stream is performed in parallel with indexing the received document of the first part of documents into the first cached indexing table; and

based on the variable arriving rate and an appropriate number of indexes to achieve an indexing performance requirement for generation of indexes, determining whether to merge the first cached indexing table and the second cached indexing table into a first receiving table for searching, or generate a second receiving table and a third receiving table respectively from the first cached indexing table and the second cached indexing table, for searching.

6. The indexing equipment of claim 5 , wherein the second cached indexing table was generated by other indexing equipment different from the indexing equipment.

7. The indexing equipment of claim 5 , wherein the operations further comprise:

merging the first cached indexing table and the second cached indexing table into a searchable index;

determining a surplus part of the first cached indexing table;

based on an amount of data corresponding to the surplus part being greater than a second threshold amount of data, merging first entries in the surplus part into second entries in the searchable index;

associating the first searchable index with documents associated with the surplus part; and

deleting the surplus part.

8. The indexing equipment of claim 7 , wherein the operations further comprise:

in response to determining that an amount of data corresponding to the searchable index is greater than a third threshold amount of data, ceasing adding to the searchable index; and

generating a new searchable index to receive entries from the first cached indexing table and the second cached indexing table.

9. A non-transitory machine-readable medium, comprising executable instructions that, when executed by a processor of an index manager device, facilitate performance of operations, comprising:

receiving respective ones of a first part of documents of an unbounded stream of documents from a document source, wherein the unbounded stream arrives at the index manager device according to a variable arriving rate;

for the respective ones of the first part of documents, generating an index request for a first index processing device, wherein the index request causes the first index processing device to index a received document of the first part of documents in a first cached indexing table generated by the first index processing device;

determining a generating speed of respective indexing requests for the first part of documents;

based on the variable arriving rate and the generating speed in relation to a first threshold, for a second part of the documents of the unbounded stream, allocating a second index processing device to perform indexing of received documents of the second part of documents in a second cached indexing table generated by the second index processing device, wherein indexing the received documents of the second part of the unbounded stream by the second index processing device is performed in parallel with the first index processing device indexing the received document of the first part of documents into the first cached indexing table; and

based on the variable arriving rate and an appropriate number of indexes to achieve an indexing performance requirement for generation of indexes, determining whether to merge the first cached indexing table and the second cached indexing table into a first receiving table for searching, or generate a second receiving table and a third receiving table respectively from the first cached indexing table and the second cached indexing table, for searching.

10. The method of claim 1 , wherein the performance requirement comprises a requirement based on an estimated size for the first receiving table.

11. The method of claim 1 , wherein the performance requirement comprises a requirement to index the unbounded stream at a selected rate.

12. The method of claim 1 , wherein the variable arriving rate in relation to the first threshold comprises the variable arriving rate being greater than the first threshold frequency of documents arriving via the unbounded stream to be indexed.

13. The method of claim 1 , wherein the performance requirement comprises a requirement that no end of the unbounded stream of documents is predicted within a period of time.

14. The indexing equipment of claim 5 , wherein the performance requirement comprises an estimated size for the first receiving table being determined to be larger than a fourth threshold.

15. The indexing equipment of claim 5 , wherein the performance requirement comprises a requirement to index the unbounded stream at a selected rate.

16. The indexing equipment of claim 5 , wherein the performance requirement comprises that no end of the stream of documents is predicted within a period of time.

17. The indexing equipment of claim 5 , wherein the operations further comprise, generating a second searchable index and a third searchable index, respectively, from the first cached index and the second cached index.

18. The non-transitory machine-readable medium of claim 9 , wherein the indexing performance requirement comprises a predicted size for the first receiving table at a predicted time the unbounded stream will stop.

19. The non-transitory machine-readable medium of claim 9 , wherein the indexing performance requirement comprises a predicted rate of indexing the unbounded stream of documents.

20. The indexing equipment of claim 5 , wherein the generating speed in relation to the first threshold comprises the generating speed being above the first threshold that was selected to adjust processing resources allocated to the unbounded stream of documents.