Document indexing with cluster computing
Methods, apparatus, and processor-readable storage media for document indexing with cluster computing are provided herein. An example computer-implemented method includes obtaining and monitoring a list of document identifiers corresponding to documents to be indexed as part of an indexing process across at least a portion of a computing cluster; initiating the indexing processing based on one or more inputs; iteratively requesting at least one of the document identifiers; obtaining, from across at least a portion of the computing cluster, a document corresponding to a document identifier provided in response to the request; and carrying out one or more functions related to the indexing process on the obtained document.
1. An apparatus comprising:
at least one processing platform comprising one or more processing devices each comprising a processor coupled to a memory;
the processing platform being configured to implement at least a portion of a computing cluster;
wherein the processing platform further comprises:
at least one indexer component resident on at least one of the one or more processing devices, wherein the at least one indexer component is configured to:
obtain and monitor a list of document identifiers corresponding to documents to be indexed as part of an indexing process across at least a portion of the computing cluster;
configure multiple source components resident on at least one of the one or more processing devices, wherein each of the multiple source components comprises a distributed actor system of multiple actors corresponding to one or more functions related to the indexing process, wherein the distributed actor system is hierarchical based at least in part on one or more communication dependencies among the multiple actors, and wherein configuring the multiple source components comprises specifying a number of instances of each actor per source component and a communication behavioral style associated with each instance; and
initiate the indexing process based on one or more inputs; and
wherein each of the multiple source components is configured to:
iteratively request at least one of the document identifiers from the at least one indexer component;
obtain, from across at least a portion of the computing cluster, a document corresponding to a document identifier provided to the source component by the at least one indexer component in response to the request; and
carry out at least one of the one or more functions related to the indexing process on the document.
2. The apparatus of claim 1 , wherein each of the multiple source components is further configured to report, to the at least one indexer component, completion of the one or more functions on the document.
3. The apparatus of claim 1 , wherein the at least one indexer component is further configured to generate the list of documents to be indexed as part of the indexing process.
4. The apparatus of claim 1 , wherein the at least one indexer component is further configured to terminate the indexing process upon completion, by the multiple source components, of the one or more functions on all of the documents to be indexed.
5. The apparatus of claim 1 , wherein the at least one indexer component is configured to monitor a status attributed to each respective one of the documents in the list during the indexing process.
6. The apparatus of claim 1 , wherein the one or more inputs comprises a user-provided schedule.
7. The apparatus of claim 1 , wherein the one or more inputs comprises a pre-determined triggering event.
8. The apparatus of claim 1 , wherein the one or more inputs comprises an on-demand user request.
9. The apparatus of claim 1 , wherein the one or more functions related to the indexing process comprises mapping one or more fields from at least a first document location to at least a second document location.
10. The apparatus of claim 1 , wherein the one or more functions related to the indexing process comprises enriching the document with one or more items of information.
11. The apparatus of claim 1 , wherein the one or more functions related to the indexing process comprises reading the document.
12. The apparatus of claim 1 , wherein the one or more functions related to the indexing process comprises saving the document, subsequent to carrying out the one or more functions, to the at least one indexer component.
13. The apparatus of claim 1 , wherein the processing platform further comprises a source component re-configuration module configured to modify the one or more functions to be carried out by the multiple source components.
14. The apparatus of claim 13 , wherein modifying the one or more functions to be carried out by the multiple source components comprises at least one of:
adding one or more additional functions to be carried out by the multiple source components;
removing one or more of the functions to be carried out by the multiple source components; and
updating at least one of one or more parameters and one or more settings for one or more existing functions.
15. A computer-implemented method comprising:
obtaining and monitoring a list of document identifiers corresponding to documents to be indexed as part of an indexing process across at least a portion of a computing cluster;
configuring multiple source components, wherein each of the multiple source components comprises a distributed actor system of multiple actors corresponding to one or more functions related to the indexing process, wherein the distributed actor system is hierarchical based at least in part on one or more communication dependencies among the multiple actors, and wherein configuring the multiple source components comprises specifying a number of instances of each actor per source component and a communication behavioral style associated with each instance;
initiating the indexing process based on one or more inputs;
iteratively requesting, via at least a portion of the multiple source components, at least one of the document identifiers;
obtaining, via at least a portion of the multiple source components, from across at least a portion of the computing cluster, a document corresponding to a document identifier provided in response to the request; and
carrying out, via at least a portion of the multiple source components, at least one of the one or more functions related to the indexing process on the obtained document;
wherein the method is performed by at least one processing device comprising a processor coupled to a memory.
16. The computer-implemented method of claim 15 , further comprising terminating the indexing process upon completion of the one or more functions on all of the documents to be indexed.
17. The computer-implemented method of claim 15 , further comprising modifying the one or more functions to be carried out.
18. A computer program product comprising a non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code when executed by a processing platform comprising a plurality of processing devices causes the processing platform:
to obtain and monitor a list of document identifiers corresponding to documents to be indexed as part of an indexing process across at least a portion of a computing cluster;
to configure multiple source components, wherein each of the multiple source components comprises a distributed actor system of multiple actors corresponding to one or more functions related to the indexing process, wherein the distributed actor system is hierarchical based at least in part on one or more communication dependencies among the multiple actors, and wherein configuring the multiple source components comprises specifying a number of instances of each actor per source component and a communication behavioral style associated with each instance;
to initiate the indexing process based on one or more inputs;
to iteratively request, via at least a portion of the multiple source components, at least one of the document identifiers;
to obtain, via at least a portion of the multiple source components, from across at least a portion of the computing cluster, a document corresponding to a document identifier provided in response to the request; and
to carry out, via at least a portion of the multiple source components, at least one of the one or more functions related to the indexing process on the obtained document.
19. The computer program product of claim 18 , wherein the program code further causes the processing platform to terminate the indexing process upon completion of the one or more functions on all of the documents to be indexed.
20. The computer program product of claim 18 , wherein the program code further causes the processing platform to modify the one or more functions to be carried out by the multiple source components.